Abstract Hand‐drawn model sheets are widely used in character design to define 3D shape and appearance through sparse multi‐view drawings. Reconstructing 3D characters from such sparse inputs has traditionally been challenging due to insufficient visual information. Recently, 3D generative models have enabled automatic reconstruction of plausible 3D characters by learning from large‐scale training data, but achieving reconstructions that accurately match the input model sheets remains limited. In this paper, we present a framework that leverages the power of a 3D generative model for initial reconstruction and enhances the output to faithfully reproduce input model sheets. For such faithful reconstruction, we must address two fundamental challenges: (1) the hand‐drawn nature inherently introduces multi‐view inconsistencies where the generated 3D geometry cannot perfectly align with all views, and (2) view‐dependent line elements along geometry boundaries interfere with accurate texture reconstruction. To address these challenges, we optimize the geometry to minimize multi‐view inconsistencies and introduce a deformable per‐pixel camera ray representation that resolves residual discrepancies in cross‐view correspondences. We also decompose drawings into three distinct layers of view‐dependent lines, view‐independent colors, and fine‐detail decals to separately handle view‐dependent and view‐independent components for consistent cross‐view reconstruction. Comprehensive experiments demonstrate that our method outperforms possible alternatives regardless of the choice of 3D generative model, while successfully preserving both artistic intent and visual fidelity of input model sheets.

This paper presents a novel framework for compactly representing a 3D indoor scene using a set of polycuboids through a deep learning-based fitting method. Indoor scenes mainly consist of man-made objects, such as furniture, which often exhibit rectilinear geometry. This property allows indoor scenes to be represented using combinations of polycuboids, providing a compact representation that benefits downstream applications like furniture rearrangement. Our framework takes a noisy point cloud as input and first detects six types of cuboid faces using a transformer network. Then, a graph neural network is used to validate the spatial relationships of the detected faces to form potential polycuboids. Finally, each polycuboid instance is reconstructed by forming a set of boxes based on the aggregated face labels. To train our networks, we introduce a synthetic dataset encompassing a diverse range of cuboid and polycuboid shapes that reflect the characteristics of indoor scenes. Our framework generalizes well to real-world indoor scene datasets, including Replica, ScanNet, and scenes captured with an iPhone. The versatility of our method is demonstrated through practical applications, such as virtual room tours and scene editing.

Abstract Non‐rigid iterative closest point (ICP) is a popular framework for shape alignment, typically formulated as alternating iteration of correspondence search and shape transformation. A common approach in the shape transformation stage is to solve a linear least squares problem to find a smoothness‐regularized transform that fits the target shape. However, completely solving the linear least squares problem to obtain a transform is wasteful because the correspondences used for constructing the problem are imperfect, especially at early iterations. In this work, we design a novel framework to compute a transform in single step without the exact linear solve. Our key idea is to use only a single step of an iterative linear system solver, conjugate gradient, at each shape transformation stage. For this single‐step scheme to be effective, appropriate preconditioning of the linear system is required. We design a novel adaptive Sobolev‐Jacobi preconditioning method for our single‐step transform to produce a large and regularized shape update suitable for correspondence search in the next iteration. We demonstrate that our preconditioned single‐step transform stably accelerates challenging 3D surface registration tasks.

Non-rigid registration of 3D shape collections using a template mesh is essential for constructing 3D datasets. Traditional non-rigid Iterative Closest Point (ICP) methods rely on manually selected template meshes, which can result in inconsistent registrations when applied to diverse shape collections. This inconsistency arises particularly when the template lacks common shape features with the input instances or when landmark annotations are sparse. To overcome this limitation, we propose a novel ICP framework that jointly optimizes a shared template shape and its instance-wise deformations. Our joint optimization framework assigns distinct roles to the shared template and instance-wise deformations: the template captures common shape features, while instance-wise deformations handle residual registration errors. We use stronger smoothness regularization on the instance-wise deformations in early iterations to prioritize the accumulation of common details on the template. Additionally, a distortion alignment energy minimizes interinstance map distortions, promoting consistent instance-wise deformations. On challenging 3D datasets with large shape variations, our method achieves state-of-the-art fitting accuracy and consistent results in shape averaging and deformation transfer. By removing the need for a carefully selected preset template, our method extends the capability of extrinsic non-rigid registration frameworks, offering a more robust and flexible solution for challenging registration scenarios.