πŸŒ„About Me

Zixun Huang is a graduate student researcher in EECS Dept at UC Berkeley, where he is fortunate to be advised by Dr. Allen Y. Yang. At FHL Vive Center, he got the opportunity leading the OpenARK team (digital twin tracking) and the 2024 NASA SUITS team (LLM enhanced human robot collaboration).

To cultivate trustworthy autonomy adept at executing complex tasks (e.g., large-scale manufacturing) in the dynamic living environments, his recent research endeavors center on advancing the realm of machine perception and geometry representation for a system-level sensing-realistic simulation and generalizable robot learning. His current research focus is on multi-modal perception and perception-rich autonomy.

Formerly, he gained experience in the AEC industry and obtained his bachelor's from Zhejiang University, where he was fortunate to engage in the robotic fabrication project of China's first all carbon fiber structure designed building. Recognized for his contributions to this project, he got the opportunity to manage a robotic fabrication lab at ZJU and assist in hosting seasonal undergrad courses and seminars to introduce robotic automation into architectural education.

πŸŽ‰Recent Highlights

12/2023. Our team has been selected to move onto Phase 2 of the NASA SUITS Challenge!

12/2023. The OpenARK team will showcase our latest progress in the 6DoF pose estimation algorithm and a robot tracking dataset at the Vive Center Gala.

10/2023. I will be giving a 2-hour online talk at Xi'an University of Architecture and Technology. The topic is "From Robotic Fabrication to 3D Scene Understanding."

10/2023. We are releasing DTTD v2! πŸŽ‰ A 3D object tracking dataset and a transformer-based 6DoF object pose estimation network that is robust to sensor depth noise.
‍
08/2023. Our paper "MARL: Multi-scale Archetype Representation Learning for Urban Building Energy Modeling" is accepted at ICCV workshop 2023, CVAAD.

01/2023. We won the MIT RealityHACK 2023 in Spatial Audio Track! πŸŽ‰

3D Vision

+ MORE

In 2022, Zixun commenced his research in 3D Vision at the FHL Vive Center for Enhanced Reality. He joined the OpenARK team under the guidance of Professor Allen Y. Yang. By 2023, he became the Lead Graduate Student Researcher and took charge of projects such as DTTD2. In addition, he supervised a team of six EECS students at UC Berkeley, guiding them to complete DTTD3. His work involved close collaboration with various camera sensors such as iPhone LiDAR, ZED, and Azure camera, encompassing algorithm research and dataset development, particularly in the areas of de-noising depth maps and 3D object pose estimation. Furthermore, he collaborated closely with PhD candidate Chenfeng on 3DΒ indoor scene reconstruction in the BAIR.

His motivation for pursuing machine perception and 3D scene understanding stemmed from his earlier studies in architecture and the AEC (Architecture, Engineering, and Construction) industry. His ultimate goal is to create a universal SDK that empowers individuals without professional expertise to easily engage in spatial computing.

πŸ“ƒRobust Digital-Twin Localization via An RGBD-based Transformer Network and A Comprehensive Evaluation on a Mobile Dataset

AI FAB

Zixun Huang*, Keling Yao*, Seth Z. Zhao*, Chuanyu Pan*, Chenfeng Xu, Kathy Zhuang, Tianjian Xu, Weiyu Feng, supervised by Dr. Allen Y. Yang
[github] [arxiv] [bibtex] [az_cam dataset] [iphone dataset] in submission to TPAMI

Are current pose estimation methods robust enough to ignore the distortion and interpolation noise in widely-adopted IPhone's LiDAR measurements? πŸ€”οΈOur DTTD-Net introduced Fourier-transform enhanced MLP and fusion-robustifying Transformer into 3D Object Tracking Tasks.

3D Gaussian Splatting

AI FAB

placeholder

DTTD3: A Comprehensive Digital-Twin Tracking Dataset Featuring Moving Robots and Diverse Depth Sensors

AI FAB

Zixun Huang*, Kathy Zhuang*, Yukun Song*, Jackson Gao*, Rui Li, Zoe Zhou, Chuanyu Pan, Seth Z. Zhao, supervised by Dr. Allen Y. Yang
[project page (under construction)]

Robustifying Pose Estimation with Generated Image Augmentation and Sparse Token Focusing

AI FAB

Zixun Huang*, Jackson Gao*, Allen Y. Yang

Robotic Fabrication

+ MORE

Zixun embarked on his journey into robotic fabrication during his undergraduate studies in 2018. His primary objective has been to establish an ongoing feedback system that bridges the gap between the design and fabrication processes. This innovation empowers architects with the actual capacity to influence on-site construction, enhancing safety for construction workers. By integrating augmented reality, robotic automation, and novel materials, he aims to foster the emergence of fresh architectural forms and styles in the digital era, unburdened by the limitations of traditional 2D drawings.

Jieyuan Shrine: Advancing Modular Architecture Construction with a Robotic Carbon-Fiber Weaving System

AI FAB

Kuanting Lai*, Zhe Liang*, Zixun Huang*, Peiyi Huang*, Yuhong Ha, Zhihao Liang, Weiwen Jian, Hyde Meng
[video] [animation] [poster] featured in: [archdaily, domus, gooood]

πŸŽ‰Our work has been featured in the most prestigious architectural media.πŸŽ‰ Β 
πŸŽ‰We achieved China’s first all carbon-fiber structure designed architecture. Density of the structure is controlled at 18KG per cubic meter and the bearing capacity of 400KG is achieved.

Efficient Discrete Construction: An Experimental Design-to-Fabrication Workflow with Automatic UAV Integration

AI FAB

Zhe Guo*, Zixun Huang*, Xuhui Lin*, Kai Xiao*, Sijie Gao*, Ziyue Hu, Xiaoliang Ying, Yitian Lu, You Lyu, Qixin Li, Lihua Zhang, Likai Wei, Hongxin Wang, Zihao Zhang, supervised by Prof. Xiang Wang*
[video] [poster] drawing credit: zczlxl3@ucl.ac.uk
❗We integrated drone tracking (via MoCap) and controlling (via ROS) into Grasshopper3D; Achieved discrete stacking with utilizing a UAV gripping system crafted from Raspberry Pi, PX4 and 3D printed hardware, etc.

πŸ“ƒA Series: Towards Mass Customization Techniques in Casting Mold Production Using 6-Axis Robotic Arms

AI FAB

Zixun Huang*, Kunshen Huang*, Zee Liang*, Sihan Wang, Weishun Xu, Prof. Raspall Felix
[IASS 2019] [SiGraDi 2020] [poster] [demo installation] [video3] This is a continuing series of researches on efficient mass customization. * denotes equal contribution. To check other students' work co-mentored by me and Zee Leong.
❗We developed a rapid 3D clay printing system and hot-wire cuttingsystem using high-torque stepper motors, Arduino, KUKA Robots, etc. πŸŽ‰I also enabled non-planar robotic printing on quadric surfaces!!

Advancing Autonomous Resin Printing in the Air: Shaping with Gravity

AI FAB

Zixun Huang, Β supervised by Dr. Dan Luo
[video] for real-world data exploration.
We achieved 3D printing in the air with the resin solidifying and being molded by gravity drop; parameterized the robotic printing behavior with extrusion speed, motion speed and dwell time, etc. Then we evaluated the capacities of imitation learning and inverse reinforcement learning on training an autonomy for the resin shaping with gravity.

Z-Bridge: Form Finding of Corrugated Concrete Shell Using Knitted Formwork

AI FAB

Binghao Yao*, Zixun Huang*, Kai Wang*, supervised by Prof. Philippe Block, Prof. Philip F. Yuan, Dr. Tom Van Mele
[project page] drawing credit: binghao_yao@berkeley.edu

We focused on shell structure optimization

Emerging

AI FAB

my role:
keywords:
tools: ARKit, PyTorch

EDUCATIONAL PROJECT
2019

Self-supervised 3D Printing with LSTM

AI FAB

github:
my role:
keywords:
tools: ARKit, PyTorch

WORKSHOP
2019
Zixun's fervor for robot learning emerged as a response to his dissatisfaction with the current state of the AEC industry, as of 2023. In this field, the researches in architectural robots typically operate within predefined trajectories due to a lack of algorithmic knowledge among practitioners and the intricate nature of on-site construction environments. His objective is to close the divide between robot learning researchers and digital fabrication researchers. He aims to draw greater attention from algorithm researchers towards applications in construction and encourage digital fabrication researchers to shift their focus from creating extravagant but impractical shapes to conducting research with more substantial societal impact.

3D Reconstruction Trajectory Planning with Camera-Pose Imitation and Off-Policy Reinforcement Learning

AI FAB

Zixun Huang*, Charlie Cheng*, Keling Yao
[project page (under construction)]

Human-AI Interaction

+ MORE

After obtaining his Bachelor's degree in Architecture in 2020, Zixun expanded his interests from architecture to urban studies. His aim was to make a more significant impact on people's lives in the near future, particularly by addressing issues of social equality. He employed a learning-based approach to analyze urban problems, making knowledge more visible to the public. Additionally, Zixun actively contributed to the development of an urban cloud platform advocating for the empowerment of diverse societal groups to have a say in designing our built environment.

His ultimate goal is to create an alternative future where everyone has the right to participate in the design of our built environment, ushering in a new design paradigm where urban designers are no longer exclusively agents of the government, and the public, regardless of professional education, can become designers of their own built environment.

πŸ“ƒDiscovering the City Through an Urban Researcher's Lens: See What They See

AI FAB

Zixun Huang, supervised by Prof. Hao Zheng, Hang Gao
[demo video] [poster] in submission to Ubicomp 2024

We are releasing "Urban Lens"!!πŸŽ‰ This work provides the public a novel way to aware our urban environment in a professional but visible way. We invited over 100 urban designers to collaborate on metrics evaluating!!

πŸ“ƒCan Machine Learning Uncover Insights into Vehicle Travel Demand from Our Built Environment?

AI FAB

Zixun Huang, Prof. Hao Zheng
[manuscript] [poster] in submission to Cities
We demonstrated that predicting people's travel demand is achievable through the observation of Points of Interest (POI) spatial distribution. πŸŽ‰Surprisingly, we found that people's activities at various times of the day are influenced by urban designβ€”an effect that extends across different cities.

Warmup: A Matrix Toolkit for Computational Photography

AI FAB

github:
keywords:
tools: ARKit, PyTorch

2022
COURSE PROJECT

Sustainability Analyser:

AI FAB

my role:
keywords:
tools: ARKit, PyTorch

2021
PROFESSIONAL PROJECT

Built Environment

+ MORE

Zixun is deeply engaged in full life cycle sustainability within the AEC industry. His work entails a thorough evaluation and enhancement of sustainability, particularly from an energy-focused standpoint, using a data-driven and learning-based approach. He was also actively involved in the development of customizable residential unit products with the aim of promoting a more environmentally friendly AEC paradigm.

πŸ“ƒMARL: Multiscale Archetype Representation Learning for Urban Building Energy Modeling

AI FAB

Xinwei Zhuang*, Zixun Huang*, Wentao Zeng, supervised by Prof. Luisa Caldas
[github] [ICCVW 2023] [bibtex] [poster]

πŸŽ‰Our work was accepted at ICCV workshop 2023, CVAAD. We achieved over 100x less computation time on urban-scale building energy estimation with significantly more accurate results.

Building Your Dream Home: Just Like Crafting with LEGO

AI FAB

Xiao Jin*, Zixun Huang*, Qianlong Zhao*, Hang Gao, Hanzhi Zhang, Qingyang Zong, supervised by Prof. Hao Zheng
[video1] [video2] [video3] executable file: [exe, apk]

We developed a modular building information management (BIM) system from 0 to 1; Enabled efficient and scalable structure customization with real-time 3D visualization; Achieved an immersive user experience built on Android using Unreal Engine and Blueprints.

Warmup: A Matrix Toolkit for Computational Photography

AI FAB

github:
keywords:
tools: ARKit, PyTorch

2022
COURSE PROJECT

Sustainability Analyser:

AI FAB

my role:
keywords:
tools: ARKit, PyTorch

2021
PROFESSIONAL PROJECT
Zixun is captivated by the capabilities of LLM and VLM for scene understanding, including tasks like Visual Question Answering (VQA), image captioning, and Scene Graph Generation (SGG). He's keen to explore how these models can excel in few-shot 3D scene understanding and its relevance in urban street scene analysis.

πŸ“ƒA Glimpse of Large Language Model Compositionality on Scene Graph Generation

AI FAB

Seth Z. Zhao*, Zixun Huang*
[project page] under construction

Prompt Learning for VQA

AI FAB

Ongoing

Ed-Bot πŸ€–οΈ: Empowering CS Students with AI-Powered Academic Support

AI FAB

Eddie Song, Tomson Qu, Zixun Huang, Yike Wang, Qing Yuan, Eric Li

Augmented Reality

+ MORE

Zixun envisions AR as a technology that will revolutionize the way people live, work, eat, and travel. To bring this vision to life, he has collaborated closely with HCI (Human-Computer Interaction) designers and game developers, engaging in the integration of prototypes and gaining a profound understanding of how algorithms can profoundly influence the world, turning science fiction into reality.

MIT RealityHACK Winner: Crafting an Immersive XR Work Environment with Spatial Audio

AI FAB

Zixun Huang*, Weiyu Feng*, Yuehui Du*, Yola Wu*, Qianchen Bao
[project page] [github]

πŸŽ‰We won the MIT RealityHACK 2023, in the track of the Best Use of Spatial Audio.
Thanks to my teammates!!! Thanking Dolby.io and Snapdragon AR for their excellent SDKs.

πŸ“ƒUniversal AR-Enhanced Interface for ROS: Enabling Multi-Type Robot Control

AI FAB

Erin Fan*, Zixun Huang*, supervised by Dr. Allen Y. Yang
[project page] under construction
This is a minimal prototype for NASA Suits 2024. Welcome to our MDes Open Showcase, we will achieve real-time Mars Rover localization and test user experiences for the LMCC (Local Mission Control Console).

Image-text Retrieval

AI FAB

Ongoing

Emerging

AI FAB

my role:
keywords:
tools: ARKit, PyTorch

EDUCATIONAL PROJECT
2019

Mat Building

AI FAB

my role: Mechanical Engineer
keywords:
tools: Drone (built with RaspberryPi 4),

2019
WORKSHOP

Self-supervised 3D Printing with LSTM

AI FAB

github:
my role:
keywords:
tools: ARKit, PyTorch

WORKSHOP
2019

πŸ™‡Find Where am I

The one in the middle holding the drone.
The one on the left with the grey mask.
The one on the right most.
Row 2; Col 3.
The one standing on the ladder.
The robotic arm and me.
Wearing the Snapdragon AR glasses.
Me with my twin sister.
Computer Vision Researcher
@FHL Vive Center, OpenARK
Focus on Millimeter-level Object Pose Estimation in AR Scenario. Sponsored by Siemens.
DATASET
‍
>> Contributed to Digital Twin Tracking Dataset (DTTD v1.0).
>> Co-first authored the DTTD v1.1 based on iPhone LiDAR camera and ARKit, in submission to WACV.
>> Achieved 44 scene annotation with over 13k frames using programmed Python & C++ toolkits.
‍
ALGORITHM
‍
>> Led the model design and training strategy of the coarse estimator for 6-Dof object tracking pipeline.
>> Achieved a coarse estimator with accuracy (ADDS 98.32%) on both YCBV & DTTD-iPhone Dataset using Transformer-based dual-encoder structure; conducted over 25 recorded ablation experiments.
02/2023
01/2023
Backend Developer &Track Winner @Spatial Audio Prize, Sponsored by Dolby.io, MIT Reality Hack 2023
Supervised by Prof. Allen Y. and worked closely with other MEng Students: Weiyu F. Seth Z. Chuanyu P. Tianjian X. Keling Y. Cui H.
Current
09/2022
3D Software Engineer, Server-end Development Lead
@INSOME Cloud
Focus on Mass Customization &Industrialization in Architectural Scenario.
APPLICATION
‍
>> Developed a building customization and management system from 0 to 1 built on Android.
>> Tools: Unreal Engine 4, Java Spring, GraphQL, Redis, Tencent Cloud, Nginx, Django 3 (Agile Prototyping), Android, MySQL
04/2022
06/2021
Product &Technology Development Leader
@Hezhu Digital
Focus on Building Energy Consumption Management &Optimization. Sponsored by Landleaf Tech.
APPLICATION
‍
>> Led and prototyped a cost management and sustainability scoring system from 0 to 1, and generalized it into launched urban planning projects.
>> Tools: Java Spring, React.js, Redux, Redis, Tencent Cloud, Nginx, Django REST Framework (Agile Prototyping), MySQL
06/2021
01/2021
Data Scientist @1DesignLab
Focus on Data-driven Urban Study. Sponsored by Alibaba Group, Chengdu Government.
03/2020
12/2019
Contract Student Design Consultant @Cainiao Network, Alibaba Group
TO B CONSULTING
>> Conducted over 85-page policy study report on logistics network business based on 15-minute city accessibility, community commerce, and revitalization of aging neighborhoods.
‍
Worked closely with Kaihang L. Weishun X. Zhiyuan X. Yunpeng W. et.al
06/2020
10/2019
Data Scientist Intern
TO G CONSULTING
‍
>> Developed Python codes to collected and analyzed about 537k online data (POI, Weibo, Twitter, MaFengWo).
>> Supported the design team to discover data-driven evidences for old towns revitalization in 3 cities: Chengdu, Xiangshan, and Xianju; produced about 400k RMB in revenue for startup design studio.
‍
Worked closely with Weishun X. et.al
12/2020
10/2019
Robotics Engineer Intern
@RoboticPlus.AI
Focus on Robotic Fabrication &Motion Planning.
ACHIEVEMENT
>> Designed and fabricated the China's first all-carbon fiber pavilion: Jieyuan Shrine and a robotic weaving system using KUKA Robots and programmable 3D modeling.
>> Achieved the 4 meters high and 3.8 meters wide entire structure weaved with a continuous line of carbon-fiber. Density of the structure is controlled at 18KG per cubic meter and the bearing capacity of 400kg is achieved.
‍
ALGORITHM
>> Planned and programmed the robotic weaving path and the simulation for 40% modules of the pavilion.
‍
Worked closely with Tim L. Zee L. Yuhong H. Peiyi H.
06/2019
03/2019
Undergrad Teaching Assistant @ZJUxRoboticsPlus Joint-course
MAIN CONTRIBUTION:
>> Improved the success rate by 63.6% of the robotic hot-wire cutting pipeline by fixing the robotic path generation algorithm.
‍
Worked closely with Hao M. Zee L. Peiyi H.
06/2019
01/2019
Architectural Intern
@Architectural Design &Research Institute of Zhejiang University Co. Ltd.
Worked closely with Zhiwei W. Han Y.
06/2018
01/2018
Current
08/2022
Master of Design, Human-AI Interation
@University of California at Berkeley
Focus on Vision &Language, Multi-modality Learning.
GPA: 4.0 (CS-related) /4.0
COMPSCI 294-196 GenAI & LLM (Ongoing)
COMPSCI 285 Reinforcement Learning(Ongoing)
COMPSCI 282 Design Deep Neural Net
COMPSCI 280 Computer Vision
COMPSCI 294-137 Immersive Computing (Audit)
COMPSCI 294-026 Computational Photography
08/2023
05/2023
XRLab Student Club @CED
ALGORITHM
>> Co-first authored an VQAE-based method for residential buildings' latent embedding and clustering.
>> Reduced the computation time by 133.7 times for NYC's residential energy consumption estimation.
‍
Worked closely with Phd. Xinwei Z.
MDes Distinguished Scholar Award
08/2022
01/2021
Research Assistant, Project Lead
@Shanghai Jiao Tong University
Focus on Machine Learning &Data-driven Urban Study.
ALGORITHM
>> First-authored an attention-based method for travel demand time-serials prediction from the perspective of built environment.
‍
APPLICATION
>> Designed and implemented an urban sense mapping system based on adversarial generation algorithms from 0 to 1.
>> Shorten the average urban quantitative survey cost from 3 days to 3 seconds per person among over 100 designers.
>> Tools: Django, React.js, PyTorch, Tencent Cloud, Nginx, uWSGI, MySQL.
07/2021
06/2021
Assistant Data Science Instructor @DigitalFUTURES World 2021, Tongji University
Worked closely with Prof. Hao Z. who currently serves in City University of Hong Kong, @Architectural Intelligence Group.
06/2020
08/2015
Bachelor of Engineering, Architecture
@Zhejiang University
Focus on Robotic Fabrication, Mass Customization, &Computational Design
My Portfolio: checkpoint-2021.pdf
Selected Course: Application of Wireless Network, Image Analysis and Processing, Fundamentals of Computer Science, C Programming, Architectural Robotics, Computer Aided Design, Parametric Design, Architectural Physics, Architectural Design, Engineering Economy, Building Codes.
‍
06/2020
03/2019
Project Lead Researcher And Teaching Assistant
COURSE NAME
>> Architectural Robotics.
>> Computational Design &Robotic Fabrication.

PUBLICATION
>> Co-first authored: Robotic Fabrication of Sustainable Hybrid Formwork with Clay and Foam for Concrete Casting.
‍
FIRST CONTRIBUTION
>> Designed the entire design-to-manufacturing workflow and led students to conduct it; wrote the first edition of the paper.
>> Engineered the entire 3D clay printing tool; achieved the SOTA efficiency for concrete formwork in material sustainable utilization, time &labor costs.
‍
Worked closely with Weishun X.
08/2019
07/2019
@Parametric Design Summer School, Tsinghua University
Focus on AI-aided Robotic Printing.
ALGORITHM
>> Conducted over 10 ablation experiments based on LSTM &MLPs to produce time-serial instructions for the extrusion speed of robotic 3D printing.
‍
Supervised by Prof. Dan L.
07/2019
06/2019
Freelance Student Mechanical Engineer @DigitalFUTURES 2019, Tongji University
MAIN CONTRIBUTION
>> Collaborated in a drones’ application onto discrete construction in 7 days with other professionals and scientists with diverse backgrounds.
‍
Supervised by Prof. Xiang W. Zhe G.
05/2019
03/2018
Student Researcher, Project Lead @Student Research Training Project
Sponsored by National Nature Science Project.
ALGORITHM
>> Achieved rapid identification and generation of convex hulls for rural settlements by designing and developing Rhino3D plug-in sisted the quantitative analysis in the national nature research project.
>> Achieved excellence award (top 10%) in SRTP.
‍
Supervised by Prof. Xincheng P.
ZJU Merit-based Scholarship
12/2016
08/2015
Sophomore @School of Materials Science and Engineering, ZJU
GPA: 3.99 (Math-related) /4.0
Selected Course: Calculus I, Calculus II, Calculus III, Linear Algebra, Ordinary Differential Equations, Inorganic &Analytical Chemistry, University Physics A, Descriptive Geometry, Engineering Graphics.