Creating relightable and animatable human characters from monocular video at a low cost is a critical task for digital human modeling and virtual reality applications. This task is complex due to intricate articulation motion, a wide range of ambient lighting conditions, and pose-dependent clothing deformations. In this paper, we introduce a novel self-supervised framework that takes a monocular video of a moving human as input and generates a 3D neural representation capable of being rendered with novel poses under arbitrary lighting conditions. Our framework decomposes dynamic humans under varying illumination into neural fields in canonical space, taking into account geometry and spatially varying BRDF material properties. Additionally, we introduce pose-driven deformation fields, enabling bidirectional mapping between canonical space and observation. Leveraging the proposed appearance decomposition and deformation fields, our framework learns in a self-supervised manner. Ultimately, based on pose-driven deformation, recovered appearance, and physically-based rendering, the reconstructed human figure becomes relightable and can be explicitly driven by novel poses. We demonstrate significant performance improvements over previous works and provide compelling examples of relighting from monocular videos of moving humans in challenging, uncontrolled capture scenarios.

Interaction using bare hands is experiencing a growing interest in mobile-based Augmented Reality (AR). Existing RGB-based works fail to provide a practical solution to identifying rich details of the hand. In this paper, we present a detection-guided method capable of recovery 3D hand posture with a color camera. The proposed method consists of key-point detectors and 3D pose optimizer. The detectors first locate the 2D hand bounding box and then apply a lightweight network on the hand region to provide a pixel-wise like-hood of hand joints. The optimizer lifts the 3D pose from the estimated 2D joints in a model-fitting manner. To ensure the result plausibly, we encode the hand shape into the objective function. The estimated 3D posture allows flexible hand-to-mobile interaction in AR applications. We extensively evaluate the proposed approach on several challenging public datasets. The experimental results indicate the efficiency and effectiveness of the proposed method.

In this paper, we propose a novel framework for hand localization and pose estimation from a single depth image. For hand localization, unlike most existing methods that using heuristic strategies, e.g. color segmentation, we propose Hierarchical Hand location Networks (HHLN) to estimate the hand location from coarse to fine in depth images, which is robust to the complex environment and efficient. It first applied at a low-resolution octree of the whole depth image and produced coarse hand region and then constructs the hand region into a high-resolution octree for fine location estimation. For pose estimation, we propose Wide Receptive-filed (WR-OCNN) which is able to capture meaningful hand structure in different scales and estimate the 3D hand pose accurately. Experiments on two widely-used hand datasets(NYU dataset and ICVL dataset) demonstrate the effectiveness and superiority of the proposed framework.

Hand pose estimation in depth images is a challenging problem for human-computer interaction. In this paper, we propose a novel approach for hand pose estimation that shares the merits of both deep learning based hand segmentation and dynamics based pose optimization. For hand segmentation, we propose `Dynamic Projected Segmentation Networks' applied at depth images, providing a pixel-wise classification result. To preserve the detailed hand-region topology structure, we design a dynamic projection based hand-region extraction method to crop the hand region from depth images. The projected hand-region is then fed into a light-weight `Encoder-Decoder Networks' for segmentation. For pose optimization, we employ rigid body dynamics to estimate the final pose based on the segmentation results which are treated as hand geometry constraints. Experiments show that our approach outperforms state-of-the-art methods on two challenging datasets.