We present 3D Points Splatting Hand Reconstruction (3D-PSHR), a real-time and photo-realistic hand reconstruction approach. We propose a self-adaptive canonical points upsampling strategy to achieve high-resolution hand geometry representation. This is followed by a self-adaptive deformation that deforms the hand from the canonical space to the target pose, adapting to the dynamic changing of canonical points which, in contrast to the common practice of subdividing the MANO model, offers greater flexibility and results in improved geometry fitting. To model texture, we disentangle the appearance color into the intrinsic albedo and pose-aware shading, which are learned through a Context-Attention module. Moreover, our approach allows the geometric and the appearance models to be trained simultaneously in an end-to-end manner. We demonstrate that our method is capable of producing animatable, photorealistic and relightable hand reconstructions using multiple datasets, including monocular videos captured with handheld smartphones and large-scale multi-view videos featuring various hand poses. We also demonstrate that our approach achieves real-time rendering speeds while simultaneously maintaining superior performance compared to existing state-of-the-art methods. • We propose 3D-PSHR, a real-time, photo-realistic hand reconstruction via point clouds. • Our method creates animatable, photorealistic, relightable hands from various datasets. • Our approach shows real-time rendering with superior performance over state-of-the-art.

Recently, deep learning based approaches have shown promising results in 3D hand reconstruction from a single RGB image. These approaches can be roughly divided into model-based approaches, which are heavily dependent on the model's parameter space, and model-free approaches, which require large numbers of 3D ground truths to reduce depth ambiguity and struggle in weakly-supervised scenarios. To overcome these issues, we propose a novel probabilistic model to achieve the robustness of model-based approaches and reduced dependence on the model's parameter space of model-free approaches. The proposed probabilistic model incorporates a model-based network as a prior-net to estimate the prior probability distribution of joints and vertices. An Attention-based Mesh Vertices Uncertainty Regression (AMVUR) model is proposed to capture dependencies among vertices and the correlation between joints and mesh vertices to improve their feature representation. We further propose a learning based occlusion-aware Hand Texture Regression model to achieve high-fidelity texture reconstruction. We demonstrate the flexibility of the proposed probabilistic model to be trained in both supervised and weakly-supervised scenarios. The experimental results demonstrate our probabilistic model's state-of-the-art accuracy in 3D hand and texture reconstruction from a single image in both training schemes, including in the presence of severe occlusions.

Deep learning for graph matching has received growing interest and developed rapidly in the past decade. Although recent deep graph matching methods have shown excellent performance on matching between graphs of equal size in the computer vision area, the size-varied graph matching problem, where the number of keypoints in the images of the same category may vary due to occlusion, is still an open and challenging problem. To tackle this, we firstly propose to formulate the combinatorial problem of graph matching as an Integer Linear Programming (ILP) problem, which is more flexible and efficient to facilitate comparing graphs of varied sizes. A novel Graph-context Attention Network (GCAN), which jointly capture intrinsic graph structure and cross-graph information for improving the discrimination of node features, is then proposed and trained to resolve this ILP problem with node correspondence supervision. We further show that the proposed GCAN model is efficient to resolve the graph-level matching problem and is able to automatically learn node-to-node similarity via graph-level matching. The proposed approach is evaluated on three public keypoint-matching datasets and one graph-matching dataset for blood vessel patterns, with experimental results showing its superior performance over existing state-of-the-art algorithms for keypoint and graph-level matching.

In this paper, we propose a novel hand-based person recognition method for the purpose of criminal investigations since the hand image is often the only available information in cases of serious crime such as sexual abuse. Our proposed method, Multi-Branch with Attention Network (MBA-Net), incorporates both channel and spatial attention modules in branches in addition to a global (without attention) branch to capture global structural information for discriminative feature learning. The attention modules focus on the relevant features of the hand image while suppressing the irrelevant backgrounds. In order to overcome the weakness of the attention mechanisms, equivariant to pixel shuffling, we integrate relative positional encodings into the spatial attention module to capture the spatial positions of pixels. Extensive evaluations on two large multi-ethnic and publicly available hand datasets demonstrate that our proposed method achieves state-of-the-art performance, surpassing the existing hand-based identification methods. The source code is available at https://github.com/nathanlem1/MBA-Net.

In many cases of serious crime, images of a hand can be the only evidence available for the forensic identification of the offender. As well as placing them at the scene, such images and video evidence offer proof of the offender committing the crime. The knuckle creases of the human hand have emerged as an effective biometric trait and been used to identify the perpetrators of child abuse in forensic investigations. However, manual utilization of knuckle creases for identification is highly time consuming and can be subjective, requiring the expertise of experienced forensic anthropologists whose availability is very limited. Hence, there arises a need for an automated approach for localization and comparison of knuckle patterns. In this paper, we present a fully automatic end-to-end approach which localizes the minor, major and base knuckles in images of the hand, and effectively uses them for identification achieving state-of-the-art results. This work improves on existing approaches and allows us to strengthen cases further by objectively combining multiple knuckles and knuckle types to obtain a holistic matching result for comparing two hands. This yields a stronger and more robust multi-unit biometric and facilitates the large-scale examination of the potential of knuckle-based identification. Evaluated on two large landmark datasets, the proposed framework achieves equal error rates (EER) of 1.0-1.9%, rank-1 accuracies of 99.3-100% and decidability indices of 5.04-5.83. We make the full results available via a novel online GUI to raise awareness with the general public and forensic investigators about the identifiability of various knuckle regions. These strong results demonstrate the value of our holistic approach to hand identification from knuckle patterns and their utility in forensic investigations.