IEEE Robotics & Automation Magazine
Special Issue on Arts and Robotics
Carnegie Mellon University
* equal contribution
Bimanual robot assemblies by BrickMatic, 10× speed
From a single natural-language prompt, Prompt-to-Product designs a physically buildable LEGO model and constructs the real product with a bimanual robot system — closing the loop from imagination to physical artifact.
Creating assembly products demands significant manual effort and expert knowledge in 1) designing the assembly and 2) constructing the product. This paper introduces Prompt-to- Product, an automated pipeline that generates real-world assembly products from natural language prompts. Specifically, we leverage LEGO bricks as the assembly platform and automate the process of creating brick assembly structures. Given the user design requirements, Prompt-to-Product generates physically buildable brick designs, and then leverages a bimanual robotic system to construct the real assembly products, bringing user imaginations into the real world. We conduct a comprehensive user study, and the results demonstrate that Prompt-to-Product significantly lowers the barrier and reduces manual effort in creating assembly products from imaginative ideas.
Prompt-to-Product decomposes the end-to-end problem into a generative designer (BrickGPT) and an assembly reasoner (BrickMatic), and hands the resulting plan to a bimanual robot for execution:
Fig. 4. BrickGPT couples an
autoregressive brick generator with a physics-aware rollback.
Given a prompt u, the model emits tokenized bricks
size (x, y, z); each step is checked
against the brick library I and a physics reasoner
S, and infeasible proposals (collisions, out-of-library)
trigger resampling until a stable design output B̂ is
produced.
Fig. 6. BrickMatic takes the design B̂, embodiment, and a robot skill set A. (a) Assembly-by-disassembly search recovers a feasible build order; (b) APEX-MR lifts that order into a task-, motion-, and action-level plan G with pick, place, and anomaly-detection skills coordinated across both arms. The plan is then executed by the bimanual system.
Interactive 3D brick designs generated by BrickGPT from user-study prompts. Drag to rotate each model.
Scroll horizontally →
BrickMatic plans and constructs the long-horizon brick assembly with closed-loop execution and error detection. Below we highlight how our system detects failure and recovers.
Scroll horizontally →
If you find our work useful, please cite:
@article{liu2025prompt2product,
title = {Prompt-to-Product: Generative Assembly via Bimanual Manipulation},
author = {Liu, Ruixuan and Huang, Philip and Pun, Ava and Deng, Kangle
and Aggarwal, Shobhit and Tang, Zhenran and Liu, Michelle
and Ramanan, Deva and Zhu, Jun-Yan and Li, Jiaoyang and Liu, Changliu},
year = {2025},
eprint = {2508.21063},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
note = {Accepted to IEEE Robotics \& Automation Magazine,
Special Issue on Arts and Robotics}
}The authors would like to thank Ken Goldberg, Oliver Kroemer, and Jean Oh for their discussions. This work is in part supported by the Manufacturing Futures Institute, Carnegie Mellon University, through a grant from the Richard King Mellon Foundation.
Prompt-to-Product builds on the following works from our group and collaborators:
ICCV 2025
An autoregressive language model that generates physically stable, buildable LEGO designs directly from natural-language descriptions — the generative designer used in Prompt-to-Product.
preprint
A high-fidelity simulator for robotic LEGO brick assembly that supports policy and planner evaluation without physical hardware.
RSS 2025
Task-, motion-, and action-level planning for two arms to cooperatively assemble a shared design — the reasoning engine behind BrickMatic.
IEEE RA-L 2024
Physics-based analysis of whether a stacked-brick design will stand under gravity and manipulation forces — the stability reasoner used inside BrickGPT.
International Symposium on Flexible Automation (ISFA), 2024
A low-cost, transferable end-effector design that lets a generic industrial arm reliably pick, place, and press-fit LEGO bricks — the hardware foundation of our bimanual system.