3D part segmentation is still an open problem in the field of 3D vision and AR/VR. Due to limited 3D labeled data, traditional supervised segmentation methods fall short in generalizing to unseen shapes and categories. Recently, the advancement in vision-language models' zero-shot abilities has brought a surge in open-world 3D segmentation methods. While these methods show promising results for$3 D$scenes or objects, they do not generalize well to 3D humans. In this paper, we present the first open-vocabulary segmentation method capable of handling 3D human. Our framework can segment the human category into desired fine-grained parts based on the textual prompt. We design a simple segmentation pipeline, leveraging SAM to generate multi-view proposals in 2D and proposing a novel Human-CLIP model to create unified embeddings for visual and textual inputs. Compared with existing pre-trained CLIP models, the HumanCLIP model yields more accurate embeddings for human-centric contents. We also design a simple-yet-effective MaskFusion module, which classifies and fuses multi-view features into 3D semantic masks without complex voting and grouping mechanisms. The design of decoupling mask proposals and text input also significantly boosts the efficiency of per-prompt inference. Experimental results on various 3D human datasets show that our method outperforms current state-of-the-art open-vocabulary 3D segmentation methods by a large margin. In addition, we showthat our method can be directly applied to various 3D representations including meshes, point clouds, and 3D Gaussian Splatting.

Quantifying the discrepancy between point sets is a critical component for point cloud learning tasks. The mainstream point cloud learning tasks utilize Chamfer distance and Earth-Mover’s distance. Chamfer distance is computationally efficient but may not fully capture differences between sets of points. Earth-Mover’s distance, while precise, is computationally expensive and can be impractical to use with high-definition data. Several variants of Sliced Wasserstein distances (SW) are introduced to reduce the computation cost, but bring new problems to the situation: The vanilla SW treats sampled slices equally, resulting in redundant projections; Distributional Sliced Wasserstein distance requires gradient-based optimization, offsetting its benefits. To overcome this limitation and leverage the advantages of Sliced Wasserstein distance over EMD, we propose a novel metric, Select-Sliced Wasserstein distance. This new distance analyzes drawn samples of slices and quantifies their informativeness for each point in a single shot, which eliminates unnecessary projections as well as costly optimizations, but perpetuates the performance. Extensive experiments on various point cloud learning tasks to demonstrate the efficiency and effectiveness of the proposed distance metric. Our code is available at https://github.com/VideoProcessingLab/SSW_Distance

3D registration is a fundamental step to obtain the correspondences between surfaces. Traditional mesh alignment methods tackle this problem through non-rigid deformation, mostly accomplished by applying ICP-based (Iterative Closest Point) optimization. The embedded deformation method is proposed for the purpose of acceleration, which enables various real-time applications. However, it regularizes on an underlying simplified structure, which could be problematic for intricate cases when the simplified graph doesn't fully represent the surface attributes. Moreover, without elaborate parameter-tuning, deformation usually performs suboptimally, leading to slow convergence or a local minimum if all regions on the surface are assumed to share the same rigidity during the optimization. In this article, we propose a novel solution that decouples regularization from the underlying deformation model by explicitly managing the rigidity of vertex clusters. We further design an efficient two-step solution that alternates between isometric deformation and embedded deformation with cluster-based regularization. Our method can easily support region-adaptive regularization with cluster refinement and execute efficiently. Extensive experiments demonstrate the effectiveness of our approach for mesh alignment tasks even under large-scale deformation and imperfect data. Our method outperforms state-of-the-art methods both numerically and visually.

Meshes generated by range scanners are often incomplete and contain complex holes due to limited input coverage and occlusion. In this paper, we present an effective method to fill the gap regions on meshes by leveraging the templates inferred from the learning-based method. We first segment both source and template models into corresponding parts. Each part will be aligned with non-rigid deformation. We then modify the gap regions by “virtual” depth maps rendered using the aligned parts from newly selected viewpoints. Comparing with the template-based mesh completion approaches, our algorithm can generate natural appearances without any user interaction. Comparing with the state-of-the-art volumetric fusion methods, our approach supports selective blending, which only modifies the regions of interest and prevents bad template inference from impacting the source.

Imagine taking a selfie video with your mobile phone and getting as output a 3D model of your head (face and 3D hair strands) that can be later used in VR, AR, and any other domain. State of the art hair reconstruction methods allow either a single photo (thus compromising 3D quality) or multiple views, but they require manual user interaction (manual hair segmentation and capture of fixed camera views that span full 360°). In this paper, we describe a system that can completely automatically create a reconstruction from any video (even a selfie video), and we don't require specific views, since taking your -90°, 90°, and full back views is not feasible in a selfie capture. In the core of our system, in addition to the automatization components, hair strands are estimated and deformed in 3D (rather than 2D as in state of the art) thus enabling superior results. We provide qualitative, quantitative, and Mechanical Turk human studies that support the proposed system, and show results on a diverse variety of videos (8 different celebrity videos, 9 selfie mobile videos, spanning age, gender, hair length, type, and styling).