This paper presents LEGO ® -Maker, a new learning-based generative model that can effectively consider over 100 unique brick types and rapidly generate hundreds of bricks to create LEGO ® models conditioned on images. This work has three major technical contributions that enable it to achieve surpassing capabilities beyond existing generative approaches. First, we design a compact LEGO ® tokenization scheme to serialize LEGO ® models and bricks into tokens for autoregressive learning. Second, we build LEGO ® -Maker, an autoregressive image-conditioned architecture, with a multi-token prediction strategy to encourage pre-considering multiple brick attributes and a rollback mechanism for collision-free generation. Third, we propose an effective data preparation pipeline with a procedural generator to synthesize LEGO ® models and a LEGO ® -to-real image translator distilled from a large vision language model to translate LEGO ® renderings into associated photorealistic images, leveraging rich prior to address the scarcity of image-to-LEGO ® data. Extensive evaluations and comparisons are conducted on two object categories, facade and portrait, over metrics in four aspects: geometry, color, semantics, and structural integrity, together with a user study. Experimental results demonstrate the versatility and compelling strengths of LEGO ® -Maker in producing structures and details given by the reference image. Also, the evaluation scores manifest that our method clearly surpasses the baselines, consistently for all evaluation metrics.

This paper presents a computational pipeline for creating personalized, physical LEGO ®1 figurines from user-input portrait photos. The generated figurine is an assembly of coherently-connected LEGO ® bricks detailed with uv-printed decals, capturing prominent features such as hairstyle, clothing style, and garment color, and also intricate details such as logos, text, and patterns. This task is non-trivial, due to the substantial domain gap between unconstrained user photos and the stylistically-consistent LEGO ® figurine models. To ensure assemble-ability by LEGO ® bricks while capturing prominent features and intricate details, we design a three-stage pipeline: (i) we formulate a CLIP-guided retrieval approach to connect the domains of user photos and LEGO ® figurines, then output physically-assemble-able LEGO ® figurines with decals excluded; (ii) we then synthesize decals on the figurines via a symmetric U-Nets architecture conditioned on appearance features extracted from user photos; and (iii) we next reproject and uv-print the decals on associated LEGO ® bricks for physical model production. We evaluate the effectiveness of our method against eight hundred expert-designed figurines, using a comprehensive set of metrics, which include a novel GPT-4V-based evaluation metric, demonstrating superior performance of our method in visual quality and resemblance to input photos. Also, we show our method's robustness by generating LEGO ® figurines from diverse inputs and physically fabricating and assembling several of them.

This paper presents the first learning-based generative pipeline for effectively creating 3D LEGO® 1 models. This task is very challenging due to the lack of dedicated representations and datasets for learning coherently-connected bricks arrangements, as well as an immense design space that is combinatorial in nature. We approach this task by focusing on creating LEGO® micro buildings. Our contributions are four-fold. First, we propose the LEGO® semantic volume representation to encode LEGO® models, considering the bricks types and bricks connections, while allowing back-propagation learning. Second, we further consider the transformative nature of LEGO ® to atomize the semantic volume and formulate a generative model to learn the representation. Third, we build a rich dataset of micro buildings for model learning. Last, we design the progressive reconstructor to create 3D LEGO® models from the generated representations, while ensuring bricks connections. We employed our pipeline to create LEGO ® micro buildings with a wide array of bricks types, demonstrating its strong capability of learning diverse micro-building styles and producing assemble-able LEGO® models. Further, we performed various quantitative evaluations, ablations, and a user study to show the compelling capability of our approach in terms of generative quality, fidelity, and diversity.

This paper presents computational methods to aid the creation of LEGO ®1 sketch models from simple input images. Beyond conventional LEGO ® mosaics, we aim to improve the expressiveness of LEGO ® models by utilizing LEGO ® tiles with sloping and rounding edges, together with rectangular bricks, to reproduce smooth curves and sharp features in the input. This is a challenging task, as we have limited brick shapes to use and limited space to place bricks. Also, the search space is immense and combinatorial in nature. We approach the task by decoupling the LEGO ® construction into two steps: first approximate the shape with a LEGO ® -buildable contour then filling the contour polygon with LEGO ® bricks. Further, we formulate this contour approximation into a graph optimization with our objective and constraints and effectively solve for the contour polygon that best approximates the input shape. Further, we extend our optimization model to handle multi-color and multi-layer regions, and formulate a grid alignment process and various perceptual constraints to refine the results. We employ our method to create a large variety of LEGO ® models and compare it with humans and baseline methods to manifest its compelling quality and speed.