We propose Splatter Layout, a single-view feedforward 3D reconstruction method that jointly predicts Gaussian splats, mesh, and point cloud in unfolded surface layout aligned with the input image. Unlike prior approaches, which suffer from inconsistent surface representations, our unfolded layout extends reconstruction to invisible regions by predicting parameters from visible neighbors and placing them near adjacent counterparts. To achieve this, we supervise the pipeline using data generated from an unfolding network, ensuring bijective mappings and input-view alignment. Since the pipeline is layout-agnostic, it can be readily extended to diverse object categories, including humans and vehicles. Without modifying the underlying architecture, Splatter Layout substantially improves the geometric fidelity of Splatter Images [42]. Its unfolded layout yields a coherent and interpretable feature organization, resolving prior inconsistencies while establishing dense correspondences with a template mesh for tasks such as animation and appearance editing.

We introduce a method to generate temporally coherent human animation from a single image, a video, or a random noise. This problem has been formulated as modeling of an auto-regressive generation, i.e., to regress past frames to decode future frames. However, such unidirectional generation is highly prone to motion drifting over time, generating unrealistic human animation with significant artifacts such as appearance distortion. We claim that bidirectional temporal modeling enforces temporal coherence on a generative network by largely suppressing the appearance ambiguity. To prove our claim, we design a novel human animation framework using a denoising diffusion model: a neural network learns to generate the image of a person by denoising temporal Gaussian noises whose intermediate results are cross-conditioned bidirectionally between consecutive frames. In the experiments, our method demonstrates strong performance compared to existing unidirectional approaches with realistic temporal coherence.

We introduce a novel implicit field representation tailored for multi-person interaction geometry in 3D spaces, capable of simultaneously reconstructing occupancy, instance identification (ID) tags, and contact fields. Volumetric representation of interacting human bodies presents significant challenges, including inaccurately captured geometries, varying degrees of occlusion, and data scarcity. Existing multi-view methods, which either reconstruct each subject in isolation or merge nearby 3D surfaces into a single unified mesh, often fail to capture the intricate geometry between interacting bodies and exploit on datasets with many views and a small group of people for training. Our approach utilizes an implicit representation for interaction geometry contextualized by a multi-view local-global feature module. This module adeptly aggregates both local and global information from individual views and interacting groups, enabling precise modeling of close physical interactions through dense point retrieval in small areas, supported by the implicit fields. Furthermore, we develop a synthetic dataset encompassing diverse multi-person interaction scenarios to enhance the robustness of our geometry estimation. The experimental results demonstrate the superiority of our method to accurately reconstruct human geometries and ID tags within three-dimensional spaces, outperforming conventional multi-view techniques. Notably, our method facilitates unsupervised estimation of contact points without the need for specific training data on contact supervision.

This paper presents the Pixel-aligned Diffusion Model (PIDiffu), a new framework for reconstructing high-fidelity clothed 3D human models from a single image. While existing PIFu variants have made significant advances using more complicated 2D and 3D feature extractions, these methods still suffer from floating artifacts and body part du-plication due to their reliance on point-wise occupancy field estimations. PIDiffu employs a diffusion-based strategy for line-wise estimation along the ray direction, conditioned by pixel-aligned features with a guided attention. This approach improves the local details and structural accuracy of the reconstructed body shape and is robust to unfamiliar and complex image features. Moreover, PIDiffu can be easily integrated with existing PIFu-based methods to leverage their advantages. The paper demonstrates that PIDiffu outperforms state-of-the-art methods that do not rely on parametric 3D body models. Especially, our method is superior in handling ’in-the-wild’ images, such as those with complex patterned clothes unseen in the training data.

Accurate gait detection is crucial in utilizing the ample health information embedded in it. Vision-based approaches for gait detection have emerged as an alternative to the exacting sensor-based approaches, but their application has been rather limited due to complicated feature engineering processes and heavy reliance on lateral views. Thus, this study aimed to find a simple vision-based approach that is view-independent and accurate. A total of 22 participants performed six different actions representing standard and peculiar gaits, and the videos acquired from these actions were used as the input of the deep learning networks. Four networks, including a 2D convolutional neural network and an attention-based deep learning network, were trained with standard gaits, and their detection performance for both standard and peculiar gaits was assessed using measures including F1-scores. While all networks achieved remarkable detection performance, the CNN-Transformer network achieved the best performance for both standard and peculiar gaits. Little deviation by the speed of actions or view angles was found. The study is expected to contribute to the wider application of vision-based approaches in gait detection and gait-based health monitoring both at home and in clinical settings.

We present a rapid and fully automatic 3D avatar creation system that can produce personalized 3D avatars within two minutes using a single depth camera and a motorized turntable. The created 3D avatar is able to make all the details of facial expressions and whole body motions including fingers. To our best knowledge, it is the first completely automatic system that can generate realistic 3D avatars in the common 3D file format, which is ready for the direct use in virtual reality applications or various services.

We present a depth-based fingertip recognition method for interactive projectors. We use a depth camera attached to a projector, so it is possible to change the relative pose between the projector and the projection surface without manual recalibration. For detection and classification of fingertips, we propose using cascaded random forests boosted by our 3-D pose-normalized pixel-difference features. The ensemble probabilities from the cascaded random forests are used to define a score function of a subset of detected fingertips. By finding the subset maximizing the score function, the fingertips in the subset are correctly classified, and the remaining incorrectly detected fingertips are rejected. Experiments show that the proposed method outperforms conventional random forest and convolutional neural network classifiers. In addition, our developed applications show the advantage of the proposed method in assigning different roles to different fingers.

Real world objects can be characterized effectively by their shape, color and material types. Material recognition of an arbitrary object at a distance is an important task for the improvement of object recognition, scene understanding, realistic rendering and various virtual and augmented reality applications. Researchers have tried to recognize material types based on color features, however material type of an object is not completely correlated with its visual appearance. In this paper, we propose a simple but effective surface roughness estimation method using single time-of-flight (ToF) camera. A set of features extracted from the estimated roughness together with conventional color features are used for material type recognition. Experimental results on our material data set with 122 subjects show promising material type recognition results.

In this paper, we present a unified approach for Expectation Maximization (EM) based motion segmentation, estimation and analysis from dense point cloud data. When identifying an underlying motion, literature mainly focuses on three related topics: motion segmentation, estimation and analysis. These topics are, however, mostly considered separately while integrated approaches are rare. Our approach specifically focuses on analyzing articulated motion from dense point cloud data by simultaneously solving for three topics using an integrated approach. No prior knowledge, such as background regions, number of segments and correspondence, is required since two iterations in this algorithm allow us to seamlessly accomplish integration of the three tasks. The first iteration of the algorithm is performed between segmentation and estimation, followed by the second iteration between motion estimation and analysis. For the first iteration, we propose EM based subspace clustering algorithm. For the second iteration, we simply fuse the motion analysis method from [1] into an iterative motion estimation algorithm. As a result, we can extract label, correspondence and motion of moving objects simultaneously from dense point cloud sequence. In experiment, we validate the performance of the proposed method on both synthetic and real world data.

We present a Generalized Deformable Spatial Pyramid (GDSP) matching algorithm for calculating the dense correspondence between a pair of images with large appearance variations. The main challenges of the problem generally originate in appearance dissimilarities and geometric variations between images. To address these challenges, we improve the existing Deformable Spatial Pyramid (DSP) [10] model by generalizing the search space and devising the spatial smoothness. The former is leveraged by rotations and scales, and the latter simultaneously considers dependencies between high-dimensional labels through the pyramid structure. Our spatial regularization in the high-dimensional space enables our model to effectively preserve the meaningful geometry of objects in the input images while allowing for a wide range of geometry variations such as perspective transform and non-rigid deformation. The experimental results on public datasets and challenging scenarios show that our method outperforms the state-of-the-art methods both qualitatively and quantitatively.

We present RGB-D Fusion, a framework which robustly tracks and reconstructs dense textured surfaces of scenes and objects by integrating both color and depth images streamed from a RGB-D sensor into a global colored volume in real-time. To handle failure of the ICP-based tracking approach, KinectFusion, due to the lack of sufficient geometric information, we propose a novel approach which registers the input RGB-D image with the colored volume by pho- tometric tracking and geometric alignment. We demonstrate the strengths of the proposed approach compared with the ICP-based approach and show superior performance of our algorithm with real-world data. I. INTRODUCTION Real-time photo-realistic 3D reconstruction of real world scenes and objects plays an important role for a variety of applications such as robotics, augmented reality, human computer interaction, and entertainment. One of the main requirements in the robot interaction with the environment is the 3D reconstruction and representation of real world environments. With the invention of the low-cost Microsoft Kinect sensor as an representative of a consumer-grade range sensing device, the high-resolution range images aligned with pho- tometric information become relatively easily available. The Kinect sensor opens up new possibilities to build a low- cost 3D modeling system with well-studied range mapping technologies.

We present a new framework that reconstructs a 3D surface by incorporating a color image into sparse depth points. Assuming that the image intensity change is highly correlated with the scene geometry, we first generate a planar mesh on the image using color variation, and then, we unproject the triangulated image to the world space by integrating the sparse depth points. A quadric error metric-based mesh simplification method is employed for effective image triangulation, and a non-linear optimization is formulated to estimate the length of the projected ray from the camera center to each vertex of the triangulated image by minimizing the errors between the reconstructed surface and the depth points. Our approach can achieve an accurate 3D surface with smooth planar regions and sharp edges on the object boundaries, and it also outperforms other surface reconstruction methods (which use only depth points) in terms of the accuracy.

Nonlocal means filtering is an edge-preserving denoising method whose filter weights are determined by Gaussian weighted patch similarities. The nonlocal means filter shows superior performance in removing additive Gaussian noise at the expense of high computational complexity. In this paper, we propose an efficient and effective denoising method by introducing a separable implementation of the nonlocal means filter and adopting a bilateral kernel for computing patch similarities. Experimental results demonstrate that the proposed method provides comparable performance to the original nonlocal means, with lower computational complexity.

We present a bi-layer inpainting method for synthesizing novel views from a single color image and its corresponding depth map under the exemplar-based inpainting framework. Unlike conventional image inpainting, the decisions of the inpainting direction and the sample regions are important to inpaint disoccluded regions which disclose hidden background regions in the new viewpoint. The proposed algorithm first labels boundaries along the disoccluded regions whether it belongs to the foreground or background objects, and then separates their surrounding regions into the foreground and background regions using the graph cut algorithm. The disoc-cluded regions are then filled from the background boundary with best-match patches taken from the background regions. As demonstrated in the experimental results, the proposed method recovers the disoccluded region with visually plausible quality.

Time-of-Flight depth cameras provide a direct way to acquire range images, using the phase delay of the incoming reflected signal with respect to the emitted signal. These cameras, however, have a challenging problem called range folding, which occurs due to the modular error in phase delay—ranges are modulo the maximum range. To our best knowledge, we exploit the first approach to estimate the number of mods at each pixel from only a single range image. The estimation is recasted into an optimization problem in the Markov random field framework, where the number of mods is considered as a label. The actual range is then recovered using the optimal number of mods at each pixel, so-named range unfolding. As demonstrated in the experiments with various range images of real scenes, the proposed method accurately determines the number of mods. In result, the maximum range is practically extended at least twice of that specified by the modulation frequency.

Human arm estimation is very important for HCI and human pose estimation, but it remains a challenging problem due to complex environments, various poses, etc. In this paper, an efficient and robust method is presented to estimate human arms from depth images. Firstly, three convex features are explored from depth images for lower/upper arm detection, including convex degree feature (CDF), convex region feature (CRF), and U-type depth feature (UDF). With these effective features, secondly, the upper arm and lower arm candidates are accurately and quickly detected in a parallel and complementary way. Finally, the full arm is estimated based on depth continuity and human configuration constraint. Experiments on lots of test images demonstrate the robustness and efficacy of this approach.

This paper presents a novel algorithm for realistic 3D face modeling from a single pair of color and time-of-flight (TOF) depth images using a generic, deformable model. Most previous approaches have attempted to emphasize either facial feature or global shape consistency, resulting in either visually plausible or geometrically accurate models. In this paper, we introduce feature-preserving surface registration to achieve both reality and accuracy. The proposed algorithm takes advantages of both facial features from the color image and geometric information from the depth image in an iterative closest point (ICP) framework, where the non-rigid registration is achieved by iteratively minimizing the energy function consisting of feature distance term, surface distance term, and linear elasticity term. As demonstrated in the experimental results, the proposed algorithm builds the personalized 3D face model within a few seconds with high visual quality.

Dynamic appearance is one of the most important cues for tracking and identifying moving people. However, direct modeling spatio-temporal variations of such appearance is often a difficult problem due to their high dimensionality and nonlinearities. In this paper we present a human tracking system that uses a dynamic appearance and motion modeling framework based on the use of robust system dynamics identification and nonlinear dimensionality reduction techniques. The proposed system learns dynamic appearance and motion models from a small set of initial frames and does not require prior knowledge such as gender or type of activity. The advantages of the proposed tracking system are illustrated with several examples where the learned dynamics accurately predict the location and appearance of the targets in future frames, preventing tracking failures due to model drifting, target occlusion and scene clutter.

This paper presents a technique to learn dynamic appearance models from a small number of training frames. Under this framework, dynamic appearance is modelled as an unknown operator that satisfies certain interpolation conditions and that can be efficiently identified using very little a priori information with off the shelf software. The advantages of the proposed method are illustrated with several examples where the leaned dynamics accurately predict the appearance of the targets preventing tracking failures due to occlusion or clutter.

A requirement common to most dynamic vision applications is the ability to track objects in a sequence of frames. This problem has been extensively studied in the past few years, leading to several techniques, such as unscented particle filter based trackers, that exploit a combination of the (assumed) target dynamics, empirically learned noise distributions and past position observations. While successful in many scenarios, these trackers remain fragile to occlusion and model uncertainty in the target dynamics. As we show in this paper, these difficulties can be addressed by modeling the dynamics of the target as an unknown operator that satisfies certain interpolation conditions. Results from interpolation theory can then be used to find this operator by solving a convex optimization problem. As illustrated with several examples, combining this operator with Kalman and UPF techniques leads to both robustness improvement and computational complexity reduction.