About Me
Zixun Huang is a research engineer at EECS Dept, UC Berkeley. He is fortunate to be advised by Dr. Allen Y. Yang at FHL Vive Center, where he started his research journey from 2022. In 2023, he got the opportunity leading the OpenARK team (digital twin tracking) and the 2024 NASA SUITS team (LLM enhanced human robot collaboration).
To cultivate trustworthy autonomy adept at executing complex tasks (e.g., building-scaled manipulation) in the dynamic living environments, his recent research endeavors center on advancing the realm of machine perception and geometry representation for a system-level sensing-realistic simulation and generalizable robot learning.
Formerly, he gained the AEC industry experience and obtained his bachelor's from Zhejiang University, where he was fortunate to engage in the project of building-scaled robotic manipulation for China's first all carbon fiber structure designed architecture, featured in ArchDaily 2019. Recognized for his contributions to this project, he got the opportunity to manage a robotic fabrication lab at ZJU and assist in hosting seasonal undergrad courses and seminars to introduce robotic automation into architectural education.
To cultivate trustworthy autonomy adept at executing complex tasks (e.g., building-scaled manipulation) in the dynamic living environments, his recent research endeavors center on advancing the realm of machine perception and geometry representation for a system-level sensing-realistic simulation and generalizable robot learning.
Formerly, he gained the AEC industry experience and obtained his bachelor's from Zhejiang University, where he was fortunate to engage in the project of building-scaled robotic manipulation for China's first all carbon fiber structure designed architecture, featured in ArchDaily 2019. Recognized for his contributions to this project, he got the opportunity to manage a robotic fabrication lab at ZJU and assist in hosting seasonal undergrad courses and seminars to introduce robotic automation into architectural education.
Recent Highlights
12/2023. Our team has been selected to move onto Phase 2 of the NASA SUITS Challenge!
12/2023. The OpenARK team will showcase our latest progress in the 6DoF pose estimation algorithm and a robot tracking dataset at the Vive Center Gala.
10/2023. I will be giving a 2-hour online talk at Xi'an University of Architecture and Technology. The topic is "From Robotic Fabrication to 3D Scene Understanding."
10/2023. We are releasing DTTD v2! A 3D object tracking dataset and a transformer-based 6DoF object pose estimation network that is robust to sensor depth noise.
β
08/2023. Our paper "MARL: Multi-scale Archetype Representation Learning for Urban Building Energy Modeling" is accepted at ICCV workshop 2023, CVAAD.
01/2023. We won the MIT RealityHACK 2023 in Spatial Audio Track!
12/2023. The OpenARK team will showcase our latest progress in the 6DoF pose estimation algorithm and a robot tracking dataset at the Vive Center Gala.
10/2023. I will be giving a 2-hour online talk at Xi'an University of Architecture and Technology. The topic is "From Robotic Fabrication to 3D Scene Understanding."
10/2023. We are releasing DTTD v2! A 3D object tracking dataset and a transformer-based 6DoF object pose estimation network that is robust to sensor depth noise.
β
08/2023. Our paper "MARL: Multi-scale Archetype Representation Learning for Urban Building Energy Modeling" is accepted at ICCV workshop 2023, CVAAD.
01/2023. We won the MIT RealityHACK 2023 in Spatial Audio Track!
My Projects
π indicates object-centric; π indicates building-scaled; * denotes equal contirbution.
Pose Estimation
πRobust Digital-Twin Localization via An RGBD-based Transformer Network and A Comprehensive Evaluation on a Mobile Dataset
AI FABZixun Huang*, Keling Yao*, Seth Z. Zhao*, Chuanyu Pan*, Chenfeng Xu, Kathy Zhuang, Tianjian Xu, Weiyu Feng, supervised by Dr. Allen Y. Yang
[github] [arxiv] [bibtex] [az_cam dataset] [iphone dataset] in submission to TPAMI
Are current pose estimation methods robust enough to ignore the distortion and interpolation noise in widely-adopted IPhone's LiDAR measurements? Our DTTD-Net introduced Fourier-transform enhanced MLP and fusion-robustifying Transformer into 3D Object Tracking Tasks.
Robotic Manipulation
π Jieyuan Shrine: Advancing Modular Architecture Construction with a Robotic Carbon-Fiber Weaving System
AI FABKuanting Lai*, Zhe Liang*, Zixun Huang*, Peiyi Huang*, Yuhong Ha, Zhihao Liang, Weiwen Jian, Hyde Meng, funded by Wutopia Lab & RoboticPlus, featured in ArchDaily
[video] [animation] [poster] [archdaily, domus, gooood]
Our work has been featured in the most prestigious architectural media. We achieved Chinaβs first all carbon-fiber structure designed architecture. Density of the structure is controlled at 18KG per cubic meter and the bearing capacity of 400KG is achieved.
ππ Efficient Discrete Construction: An Experimental Design-to-Fabrication Workflow with Automatic UAV Integration
AI FABZhe Guo*, Zixun Huang*, Xuhui Lin*, Kai Xiao*, Sijie Gao*, Ziyue Hu, Xiaoliang Ying, Yitian Lu, You Lyu, Qixin Li, Lihua Zhang, Likai Wei, Hongxin Wang, Zihao Zhang, supervised by Prof. Xiang Wang*
[video] [poster] drawing credit: zczlxl3@ucl.ac.uk
We integrated drone tracking (via MoCap) and controlling (via ROS) into Grasshopper3D; Achieved discrete stacking with utilizing a UAV gripping system crafted from Raspberry Pi, PX4 and 3D printed hardware, etc.
π A Series: Towards Mass Customization Techniques in Casting Mold Production Using 6-Axis Robotic Arms
AI FABZixun Huang*, worked closely in a series of projects with Kunshen Huang*, Zee Liang*, Sihan Wang, Weishun Xu, Prof. Raspall Felix
[IASS 2019] [SiGraDi 2020] [poster] [demo installation] [video3] This is a
continuing series of researches on efficient mass customization. * denotes equal contribution considering multiple projects. To check other
students' work co-mentored by me and Zee Leong.
We developed a rapid 3D clay printing system and hot-wire cuttingsystem using high-torque stepper motors, Arduino, KUKA Robots, etc. I also enabled non-planar robotic printing on quadric surfaces!!
View
Project
Project
<Waiting>Travel Demand & Built Environment:
AI FABML Engineering
View
Project
Project
XR Working Environment Editor: MIT Reality HACK 2023 Winner
AI FABAR & Andriod Dev
View
Project
Project
INSOME Cloud:
AI FABAndriod Dev
View
Project
Project
Generative Adversarial City Sense Generation
AI FABWeb Dev
Find Where am I
The one in the middle holding the drone.
The one on the left with the grey mask.
The one on the right most.
Row 2; Col 3.
The one standing on the ladder.
The robotic arm and me.
Wearing the Snapdragon AR glasses.
Me with my twin sister.
Computer Vision Researcher
@FHL Vive Center, OpenARK
Focus on Millimeter-level Object Pose Estimation in AR Scenario. Sponsored by Siemens.
DATASET
β>> Contributed to Digital Twin Tracking Dataset (DTTD v1.0).
>> Co-first authored the DTTD v1.1 based on iPhone LiDAR camera and ARKit, in submission to WACV.
>> Achieved 44 scene annotation with over 13k frames using programmed Python & C++ toolkits.
β
ALGORITHM
β>> Led the model design and training strategy of the coarse estimator for 6-Dof object tracking pipeline.
>> Achieved a coarse estimator with accuracy (ADDS 98.32%) on both YCBV & DTTD-iPhone Dataset using Transformer-based dual-encoder structure; conducted over 25 recorded ablation experiments.
β>> Contributed to Digital Twin Tracking Dataset (DTTD v1.0).
>> Co-first authored the DTTD v1.1 based on iPhone LiDAR camera and ARKit, in submission to WACV.
>> Achieved 44 scene annotation with over 13k frames using programmed Python & C++ toolkits.
β
ALGORITHM
β>> Led the model design and training strategy of the coarse estimator for 6-Dof object tracking pipeline.
>> Achieved a coarse estimator with accuracy (ADDS 98.32%) on both YCBV & DTTD-iPhone Dataset using Transformer-based dual-encoder structure; conducted over 25 recorded ablation experiments.
02/2023
01/2023
Backend Developer &Track Winner @Spatial Audio Prize, Sponsored by Dolby.io, MIT Reality Hack 2023
Supervised by Prof. Allen Y. and worked closely with other MEng Students: Weiyu F. Seth Z. Chuanyu P. Tianjian X. Keling Y. Cui H.
Current
09/2022
3D Software Engineer, Server-end Development Lead
@INSOME Cloud
Focus on Mass Customization &Industrialization in Architectural Scenario.
APPLICATION
β>> Developed a building customization and management system from 0 to 1 built on Android.
>> Tools: Unreal Engine 4, Java Spring, GraphQL, Redis, Tencent Cloud, Nginx, Django 3 (Agile Prototyping), Android, MySQL
β>> Developed a building customization and management system from 0 to 1 built on Android.
>> Tools: Unreal Engine 4, Java Spring, GraphQL, Redis, Tencent Cloud, Nginx, Django 3 (Agile Prototyping), Android, MySQL
04/2022
06/2021
Product &Technology Development Leader
@Hezhu Digital
Focus on Building Energy Consumption Management &Optimization. Sponsored by Landleaf Tech.
APPLICATION
β>> Led and prototyped a cost management and sustainability scoring system from 0 to 1, and generalized it into launched urban planning projects.
>> Tools: Java Spring, React.js, Redux, Redis, Tencent Cloud, Nginx, Django REST Framework (Agile Prototyping), MySQL
β>> Led and prototyped a cost management and sustainability scoring system from 0 to 1, and generalized it into launched urban planning projects.
>> Tools: Java Spring, React.js, Redux, Redis, Tencent Cloud, Nginx, Django REST Framework (Agile Prototyping), MySQL
06/2021
01/2021
Data Scientist @1DesignLab
Focus on Data-driven Urban Study. Sponsored by Alibaba Group, Chengdu Government.
03/2020
12/2019
Contract Student Design Consultant @Cainiao Network, Alibaba Group
TO B CONSULTING
>> Conducted over 85-page policy study report on logistics network business based on 15-minute city accessibility, community commerce, and revitalization of aging neighborhoods.
β
Worked closely with Kaihang L. Weishun X. Zhiyuan X. Yunpeng W. et.al
>> Conducted over 85-page policy study report on logistics network business based on 15-minute city accessibility, community commerce, and revitalization of aging neighborhoods.
β
Worked closely with Kaihang L. Weishun X. Zhiyuan X. Yunpeng W. et.al
06/2020
10/2019
Data Scientist Intern
TO G CONSULTING
β>> Developed Python codes to collected and analyzed about 537k online data (POI, Weibo, Twitter, MaFengWo).
>> Supported the design team to discover data-driven evidences for old towns revitalization in 3 cities: Chengdu, Xiangshan, and Xianju; produced about 400k RMB in revenue for startup design studio.
β
Worked closely with Weishun X. et.al
β>> Developed Python codes to collected and analyzed about 537k online data (POI, Weibo, Twitter, MaFengWo).
>> Supported the design team to discover data-driven evidences for old towns revitalization in 3 cities: Chengdu, Xiangshan, and Xianju; produced about 400k RMB in revenue for startup design studio.
β
Worked closely with Weishun X. et.al
12/2020
10/2019
Robotics Engineer Intern
@RoboticPlus.AI
Focus on Robotic Fabrication &Motion Planning.
ACHIEVEMENT
>> Designed and fabricated the China's first all-carbon fiber pavilion: Jieyuan Shrine and a robotic weaving system using KUKA Robots and programmable 3D modeling.
>> Achieved the 4 meters high and 3.8 meters wide entire structure weaved with a continuous line of carbon-fiber. Density of the structure is controlled at 18KG per cubic meter and the bearing capacity of 400kg is achieved.
β
ALGORITHM
>> Planned and programmed the robotic weaving path and the simulation for 40% modules of the pavilion.
β
Worked closely with Tim L. Zee L. Yuhong H. Peiyi H.
>> Designed and fabricated the China's first all-carbon fiber pavilion: Jieyuan Shrine and a robotic weaving system using KUKA Robots and programmable 3D modeling.
>> Achieved the 4 meters high and 3.8 meters wide entire structure weaved with a continuous line of carbon-fiber. Density of the structure is controlled at 18KG per cubic meter and the bearing capacity of 400kg is achieved.
β
ALGORITHM
>> Planned and programmed the robotic weaving path and the simulation for 40% modules of the pavilion.
β
Worked closely with Tim L. Zee L. Yuhong H. Peiyi H.
06/2019
03/2019
Undergrad Teaching Assistant @ZJUxRoboticsPlus Joint-course
MAIN CONTRIBUTION:
>> Improved the success rate by 63.6% of the robotic hot-wire cutting pipeline by fixing the robotic path generation algorithm.
β
Worked closely with Hao M. Zee L. Peiyi H.
>> Improved the success rate by 63.6% of the robotic hot-wire cutting pipeline by fixing the robotic path generation algorithm.
β
Worked closely with Hao M. Zee L. Peiyi H.
06/2019
01/2019
Current
08/2022
Master of Design, Human-AI Interation
@University of California at Berkeley
Focus on Vision &Language, Multi-modality Learning.
GPA: 4.0 (CS-related) /4.0
COMPSCI 294-196 GenAI & LLM (Ongoing)
COMPSCI 285 Reinforcement Learning(Ongoing)
COMPSCI 282 Design Deep Neural Net
COMPSCI 280 Computer Vision
COMPSCI 294-137 Immersive Computing (Audit)
COMPSCI 294-026 Computational Photography
COMPSCI 294-196 GenAI & LLM (Ongoing)
COMPSCI 285 Reinforcement Learning(Ongoing)
COMPSCI 282 Design Deep Neural Net
COMPSCI 280 Computer Vision
COMPSCI 294-137 Immersive Computing (Audit)
COMPSCI 294-026 Computational Photography
08/2023
05/2023
XRLab Student Club @CED
ALGORITHM
>> Co-first authored an VQAE-based method for residential buildings' latent embedding and clustering.
>> Reduced the computation time by 133.7 times for NYC's residential energy consumption estimation.
β
Worked closely with Phd. Xinwei Z.
>> Co-first authored an VQAE-based method for residential buildings' latent embedding and clustering.
>> Reduced the computation time by 133.7 times for NYC's residential energy consumption estimation.
β
Worked closely with Phd. Xinwei Z.
MDes Distinguished Scholar Award
08/2022
01/2021
Research Assistant, Project Lead
@Shanghai Jiao Tong University
Focus on Machine Learning &Data-driven Urban Study.
ALGORITHM
>> First-authored an attention-based method for travel demand time-serials prediction from the perspective of built environment.
β
APPLICATION
>> Designed and implemented an urban sense mapping system based on adversarial generation algorithms from 0 to 1.
>> Shorten the average urban quantitative survey cost from 3 days to 3 seconds per person among over 100 designers.
>> Tools: Django, React.js, PyTorch, Tencent Cloud, Nginx, uWSGI, MySQL.
>> First-authored an attention-based method for travel demand time-serials prediction from the perspective of built environment.
β
APPLICATION
>> Designed and implemented an urban sense mapping system based on adversarial generation algorithms from 0 to 1.
>> Shorten the average urban quantitative survey cost from 3 days to 3 seconds per person among over 100 designers.
>> Tools: Django, React.js, PyTorch, Tencent Cloud, Nginx, uWSGI, MySQL.
07/2021
06/2021
Assistant Data Science Instructor @DigitalFUTURES World 2021, Tongji University
Worked closely with Prof. Hao Z. who currently serves in City University of Hong Kong, @Architectural Intelligence Group.
06/2020
08/2015
Bachelor of Engineering, Architecture
@Zhejiang University
Focus on Robotic Fabrication, Mass Customization, &Computational Design
My Portfolio: checkpoint-2021.pdf
Selected Course: Application of Wireless Network, Image Analysis and Processing, Fundamentals of Computer Science, C Programming, Architectural Robotics, Computer Aided Design, Parametric Design, Architectural Physics, Architectural Design, Engineering Economy, Building Codes.
β
Selected Course: Application of Wireless Network, Image Analysis and Processing, Fundamentals of Computer Science, C Programming, Architectural Robotics, Computer Aided Design, Parametric Design, Architectural Physics, Architectural Design, Engineering Economy, Building Codes.
β
06/2020
03/2019
Project Lead Researcher And Teaching Assistant
COURSE NAME
>> Architectural Robotics.
>> Computational Design &Robotic Fabrication.
PUBLICATION
>> Co-first authored: Robotic Fabrication of Sustainable Hybrid Formwork with Clay and Foam for Concrete Casting.
β
FIRST CONTRIBUTION
>> Designed the entire design-to-manufacturing workflow and led students to conduct it; wrote the first edition of the paper.
>> Engineered the entire 3D clay printing tool; achieved the SOTA efficiency for concrete formwork in material sustainable utilization, time &labor costs.
β
Worked closely with Weishun X.
>> Architectural Robotics.
>> Computational Design &Robotic Fabrication.
PUBLICATION
>> Co-first authored: Robotic Fabrication of Sustainable Hybrid Formwork with Clay and Foam for Concrete Casting.
β
FIRST CONTRIBUTION
>> Designed the entire design-to-manufacturing workflow and led students to conduct it; wrote the first edition of the paper.
>> Engineered the entire 3D clay printing tool; achieved the SOTA efficiency for concrete formwork in material sustainable utilization, time &labor costs.
β
Worked closely with Weishun X.
08/2019
07/2019
@Parametric Design Summer School, Tsinghua University
Focus on AI-aided Robotic Printing.
07/2019
06/2019
Freelance Student Mechanical Engineer @DigitalFUTURES 2019, Tongji University
05/2019
03/2018
Student Researcher, Project Lead @Student Research Training Project
Sponsored by National Nature Science Project.
ZJU Merit-based Scholarship
12/2016
08/2015
Sophomore @School of Materials Science and Engineering, ZJU
GPA: 3.99 (Math-related) /4.0
Selected Course: Calculus I, Calculus II, Calculus III, Linear Algebra, Ordinary Differential Equations, Inorganic &Analytical Chemistry, University Physics A, Descriptive Geometry, Engineering Graphics.
Selected Course: Calculus I, Calculus II, Calculus III, Linear Algebra, Ordinary Differential Equations, Inorganic &Analytical Chemistry, University Physics A, Descriptive Geometry, Engineering Graphics.
