Existing human pose estimation methods seldom consider the impact or constraint of different types of contact. In this paper, we elaborate on the impact of both body-scene contact and self-contact on pose estimation and refer to them as general contact. First, we extend existing datasets by calculating additional contact labels for general contact inference. Moreover, based on the extended dataset, we present the first network to predict dense general contact from a single RGB image. Finally, we develop a novel optimization method that successfully utilizes the inferred general contact information for accurate 3D pose estimation. Our results show that knowledge of contact can provide strong constraints and resolve pose ambiguity, thus significantly improving human pose estimation accuracy, especially for challenging poses that cannot be well handled by existing methods. Experimental results and comparisons further demonstrate the effectiveness of the proposed method. Our results are even more reasonable than certain pseudo-ground truth determined from multi-view images.

Foot contact is an important cue for human motion capture, understanding, and generation. Existing datasets tend to annotate dense foot contact using visual matching with thresholding or incorporating pressure signals. However, these approaches either suffer from low accuracy or are only designed for small-range and slow motion. There is still a lack of a vision-pressure multimodal dataset with large-range and fast human motion, as well as accurate and dense foot-contact annotation. To fill this gap, we propose a Multimodal MoCap Dataset with Vision and Pressure sensors, named MMVP. MMVP provides accurate and dense plantar pressure signals synchronized with RGBD observations, which is especially useful for both plausible shape estimation, robust pose fitting without foot drifting, and accurate global translation tracking. To validate the dataset, we propose an RGBD-P SMPL fitting method and also a monocular-video-based baseline framework, VP-MoCap, for human motion capture. Experiments demonstrate that our RGBD-P SMPL Fitting results significantly outperform pure visual motion capture. Moreover, VP-MoCap outperforms SOTA methods in foot-contact and global translation estimation accuracy. We believe the configuration of the dataset and the baseline frameworks will stimulate the research in this direction and also provide a good reference for MoCap applications in various domains. Project page: https://metaverse-ai-lab-thu.github.io/MMVP-Dataset/

The existing infrared (IR) sea-sky line (SSL) detection algorithms usually treat the SSL as a straight line, therefore cannot handle the curved SSL caused by lens distortion and earth curvature. To address this issue, a simple but efficient one-stage end-to-end network termed CSSLNet that is specially designed for curved IR SSL detection is proposed, which treats the SSL as a curve and achieves accurate detection by directly predicting the coefficients of the polynomial. Firstly, by predicting the fitting coefficients, a quadratic polynomial is obtained to describe the curved SSL. Then, to solve the sway problem of the curve, the intersection points of the SSL and image boundaries are used as additional weights to provide an overall constraint of the curve. Finally, to solve the problem of insufficient training samples of the curved SSL, an image augmentation method is employed to enlarge the dataset by performing warping operations on real IR SSL images. The experiments based on the dataset which includes natural straight, natural curve, and artificially curved SSL indicated that the proposed algorithm could achieve precise detection on both straight and curved SSL, which is comparable with the state-of-the-art in precision and speed. Moreover, the CSSLNet is the first algorithm that is specially designed for curved SSL detection, which can be a baseline for future research.

Fast and accurate infrared (IR) sea–sky line (SSL) detection could greatly benefit the efficiency of target detection under maritime. However, the traditional SSL detection algorithms are slightly inferior in detection accuracy, while the convolutional neural network (CNN)-based algorithms have high standard hardware and dataset requirements, which are difficult to satisfy in some practical scenes. In this article, a novel concise and intuitive SSL detection algorithm named bisection window and homogeneity principle-based local contrast measure (BHLCM) is proposed. First, a bisection local contrast window (BLCW) is proposed based on the local contrast measure (LCM) and be used for the search of the patches that contain SSL segments along a set of variable preset vertical paths. Then, based on the analysis of the physical characteristics of areas near SSL, the homogeneity principle is proposed to remove the patches containing false SSL segments. Finally, the midpoints of the remaining SSL segments are used to fit the final SSL through random sample consensus (RANSAC). Experimental results based on four IR image sequences with more than 3000 IR images illustrate that compared with the state-of-the-art algorithms, BHLCM not only achieves precision comparable to the most accurate one but also is significantly ahead in speed among most of the algorithms compared. In addition, speed tests about BHLCM based on different hardware platforms show that real-time detection could still be achieved even without high-performance graphics cards. The code and dataset are available at BHLCMhttps://github.com/FJsRepo/BHLCM.

Infrared (IR) small dim target detection near the sea-sky line (SSL) is crucial for enhancing the early warning capability of maritime vehicles. However, the interferences caused by the strong contrast have not been properly addressed. Consequently, a specially designed algorithm Regional-Division Local Contrast Measure (RDLCM) that focuses on the detection of infrared small dim targets appearing near the SSL is proposed. First, an SSL detection module based on a lightweight Convolutional Neural Network (CNN) is devised to achieve fast pixel-level SSL detection. Then, a set of regional-division windows (RDW) are designed according to the strong grayscale contrast distribution around the SSL, through the division of the effective regions, the RDWs could realize the potential extraction and refinement of the IR small dim targets that appear near the SSL. Experiments on three IR image sequences demonstrate that the proposed algorithm achieves the best detection accuracy among the classical and the state-of-the-art algorithms in comparison, and runs at 44 frames per second (FPS) which could meet real-time requirements. The code and dataset are available at RDLCM.

In this paper, we propose a controllable high-quality free viewpoint video generation method based on the motion graph and neural radiance fields (NeRF). Different from existing pose-driven NeRF or time/structure conditioned NeRF works, we propose to first construct a directed motion graph of the captured sequence. Such a sequence-motion-parameterization strategy not only enables flexible pose control for free viewpoint video rendering but also avoids redundant calculation of similar poses and thus improves the overall reconstruction efficiency. Moreover, to support body shape control without losing the realistic free viewpoint rendering performance, we improve the vanilla NeRF by combining explicit surface deformation and implicit neural scene representations. Specifically, we train a local surface-guided NeRF for each valid frame on the motion graph, and the volumetric rendering was only performed in the local space around the real surface, thus enabling plausible shape control ability. As far as we know, our method is the first method that supports both realistic free viewpoint video reconstruction and motion graph-based user-guided motion traversal. The results and comparisons further demonstrate the effectiveness of the proposed method.

Infrared (IR) sea-sky line (SSL) detection is an important assisting method for maritime monitoring. In complex sea-sky backgrounds, strong clouds and waves often have IR features similar to the real SSL, which challenges the true positive rate (TPR) of SSL detection. We propose a new SSL detection algorithm based on a novel self-adaptive Laplacian of Gaussian (SALoG) filter and visual-saliency-based probabilistic Hough transform (VSBPHT). The Laplacian filter of the SALoG has a self-adaptive new kernel, which is calculated by the overall gray information of the image. Its proportional-gray-reduction feature can weaken the interferences, especially in the sky region, and relatively enhance the SSL simultaneously, so that we can detect the SSL from the top to the bottom of the image which could avoid the disturbances in the complex sea region. Also, in order to improve the processing speed and accuracy, the algorithm uses probabilistic Hough transform (PHT) to extract the candidate line segment set and refine it by SSL visual saliency features, the final segments are then used to fit the SSL. Experimental results based on three IR image sequences with different backgrounds show that the proposed algorithm has the highest TPR and precision compared with state-of-the-art algorithms, and the average time consumption could meet the needs for real-time processing as well.

We propose DoubleFusion, a new real-time system that combines volumetric non-rigid reconstruction with data-driven template fitting to simultaneously reconstruct detailed surface geometry, large non-rigid motion and the optimized human body shape from a single depth camera. One of the key contributions of this method is a double-layer representation consisting of a complete parametric body model inside, and a gradually fused detailed surface outside. A pre-defined node graph on the body parameterizes the non-rigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double-layer representation to enable robust and fast motion tracking performance. Moreover, the inner parametric body is optimized online and forced to fit inside the outer surface layer as well as the live depth input. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. Experiments and comparisons show improved fast motion tracking and loop closure performance on more challenging scenarios. Two extended applications including body measurement and shape retargeting show the potential of our system in terms of practical use.

This paper proposes a new method for live free-viewpoint human performance capture with dynamic details (e.g., cloth wrinkles) using a single RGBD camera. Our main contributions are: (i) a multi-layer representation of garments and body, and (ii) a physics-based performance capture procedure. We first digitize the performer using multi-layer surface representation, which includes the undressed body surface and separate clothing meshes. For performance capture, we perform skeleton tracking, cloth simulation, and iterative depth fitting sequentially for the incoming frame. By incorporating cloth simulation into the performance capture pipeline, we can simulate plausible cloth dynamics and cloth-body interactions even in the occluded regions, which was not possible in previous capture methods. Moreover, by formulating depth fitting as a physical process, our system produces cloth tracking results consistent with the depth observation while still maintaining physical constraints. Results and evaluations show the effectiveness of our method. Our method also enables new types of applications such as cloth retargeting, free-viewpoint video rendering and animations.

We propose DoubleFusion, a new real-time system that combines volumetric dynamic reconstruction with data-driven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside, and a gradually fused outer surface layer. A pre-defined node graph on the body surface parameterizes the non-rigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.

We propose BodyFusion, a novel real-time geometry fusion method that can track and reconstruct non-rigid surface motion of a human performance using a single consumer-grade depth camera. To reduce the ambiguities of the non-rigid deformation parameterization on the surface graph nodes, we take advantage of the internal articulated motion prior for human performance and contribute a skeleton-embedded surface fusion (SSF) method. The key feature of our method is that it jointly solves for both the skeleton and graph-node deformations based on information of the attachments between the skeleton and the graph nodes. The attachments are also updated frame by frame based on the fused surface geometry and the computed deformations. Overall, our method enables increasingly denoised, detailed, and complete surface reconstruction as well as the updating of the skeleton and attachments as the temporal depth frames are fused. Experimental results show that our method exhibits substantially improved nonrigid motion fusion performance and tracking robustness compared with previous state-of-the-art fusion methods. We also contribute a dataset for the quantitative evaluation of fusion-based dynamic scene reconstruction algorithms using a single depth camera.