Estimating the 3D poses of hands and objects from a single RGB image is a fundamental yet challenging problem, with broad applications in augmented reality and human-computer interaction. Existing methods largely rely on visual cues alone, often producing results that violate physical constraints such as interpenetration or non-contact. Recent efforts to incorporate physics reasoning typically depend on post-optimization or non-differentiable physics engines, which compromise visual consistency and end-to-end trainability. To overcome these limitations, we propose a novel framework that jointly integrates visual and physical cues for hand-object pose estimation. This integration is achieved through two key ideas: 1) joint visual-physical cue learning: The model is trained to extract 2D visual cues and 3D physical cues, thereby enabling more comprehensive representation learning for hand-object interactions; 2) candidate pose aggregation: A novel refinement process that aggregates multiple diffusion-generated candidate poses by leveraging both visual and physical predictions, yielding a final estimate that is visually consistent and physically plausible. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches in both pose accuracy and physical plausibility.

In hand pose estimation, challenges such as occlusion often result in partial observation of a human hand, making it difficult to uniquely determine the hand pose, thus leading to ambiguity in certain hand regions. Heatmap-based methods may struggle with locating ambiguous joints and end up violating physiological constraints in their predictions. Parametric model based single-solution methods often fail to adequately address this ambiguity issue due to the inherent one-to-many mappings between input and output, resulting in unstable regression. While some existing multi-hypothesis methods have improved diversity by directly modeling the distribution of ambiguous hypotheses, their localization accuracy still falls short compared to the recent single-solution methods. To achieve quality results in both diversity and accuracy, we propose a novel multi-hypothesis approach for hand pose estimation, by progressively integrating heatmap information into the distribution of ambiguous poses using a RANSAC-like strategy. It starts with a conditional-flow model to provide an initial estimate of a coarse distribution over ambiguous joint poses. This is followed by randomly sampling multiple hypotheses, projecting each of them onto 2D heatmap plane, and employing consensus checks to identify unambiguous joints that adhere to skeletal constraints. Joint features are then resampled, with mismatches due to incorrect estimations being eliminated. Finally, we refine the distribution of ambiguous poses using graph neural networks and attention mechanisms. Extensive empirical experiments are carried out, where our approach are carefully examined both qualitatively and quantitatively. It is shown to not only produce more diverse & feasible pose hypotheses than existing multi-hypothesis methods, but also achieves accurate localization results comparable to the state-of-the-art single-solution methods.

The training of depth image-based hand pose estimation model typically relies on real-life datasets which are expected to be 1) largescale and cover a diverse range of hand poses and hand shapes, and 2) always come with high-precision annotations. However, existing datasets in reality are rather limited in the above regards due to multitude practical constraints, with time and cost being the major concerns. This observation motivates us to propose an alternative approach, where hand pose model is primarily trained with synthesized hand depth images that closely mimicking the characteristic noise patterns of a specific depth camera make under consideration. It is achieved by firstly mapping a Gaussian distributed variable to certain specific non-i.i.d. (independent and identically distributed) depth noise pattern, and then transforming a “vanilla” noise-free synthetic depth image to a realistic-looking image. Extensive empirical experiments demonstrate that our approach is capable of generating camera-specific realistic-looking hand depth images with precise annotations; comparing to entirely relying on annotated real images, a hand pose model with better performance is obtained by using only a small fraction (10%) of annotated real images as well as our synthesized images.