Abstract Hand‐drawn model sheets are widely used in character design to define 3D shape and appearance through sparse multi‐view drawings. Reconstructing 3D characters from such sparse inputs has traditionally been challenging due to insufficient visual information. Recently, 3D generative models have enabled automatic reconstruction of plausible 3D characters by learning from large‐scale training data, but achieving reconstructions that accurately match the input model sheets remains limited. In this paper, we present a framework that leverages the power of a 3D generative model for initial reconstruction and enhances the output to faithfully reproduce input model sheets. For such faithful reconstruction, we must address two fundamental challenges: (1) the hand‐drawn nature inherently introduces multi‐view inconsistencies where the generated 3D geometry cannot perfectly align with all views, and (2) view‐dependent line elements along geometry boundaries interfere with accurate texture reconstruction. To address these challenges, we optimize the geometry to minimize multi‐view inconsistencies and introduce a deformable per‐pixel camera ray representation that resolves residual discrepancies in cross‐view correspondences. We also decompose drawings into three distinct layers of view‐dependent lines, view‐independent colors, and fine‐detail decals to separately handle view‐dependent and view‐independent components for consistent cross‐view reconstruction. Comprehensive experiments demonstrate that our method outperforms possible alternatives regardless of the choice of 3D generative model, while successfully preserving both artistic intent and visual fidelity of input model sheets.

Non-rigid registration of 3D shape collections using a template mesh is essential for constructing 3D datasets. Traditional non-rigid Iterative Closest Point (ICP) methods rely on manually selected template meshes, which can result in inconsistent registrations when applied to diverse shape collections. This inconsistency arises particularly when the template lacks common shape features with the input instances or when landmark annotations are sparse. To overcome this limitation, we propose a novel ICP framework that jointly optimizes a shared template shape and its instance-wise deformations. Our joint optimization framework assigns distinct roles to the shared template and instance-wise deformations: the template captures common shape features, while instance-wise deformations handle residual registration errors. We use stronger smoothness regularization on the instance-wise deformations in early iterations to prioritize the accumulation of common details on the template. Additionally, a distortion alignment energy minimizes interinstance map distortions, promoting consistent instance-wise deformations. On challenging 3D datasets with large shape variations, our method achieves state-of-the-art fitting accuracy and consistent results in shape averaging and deformation transfer. By removing the need for a carefully selected preset template, our method extends the capability of extrinsic non-rigid registration frameworks, offering a more robust and flexible solution for challenging registration scenarios.