We present ViTaM-D, a novel visual-tactile framework for reconstructing dynamic hand-object interaction with distributed tactile sensing to enhance contact modeling. Existing methods, relying solely on visual inputs, often fail to capture occluded interactions and object deformation. To address this, we introduce DF-Field, a distributed force-aware contact representation leveraging kinetic and potential energy in hand-object interactions. ViTaM-D first reconstructs interactions using a visual network with contact constraint, then refines contact details through force-aware optimization, improving object deformation modeling. To evaluate deformable object reconstruction, we introduce the HOT dataset, featuring 600 hand-object interaction sequences in a high-precision simulation environment. Experiments on DexYCB and HOT datasets show that ViTaM-D outperforms state-of-the-art methods in reconstruction accuracy for both rigid and deformable objects. DF-Field also proves more effective in refining hand poses and enhancing contact modeling than previous refinement methods. The code, models, and datasets are available at https://sites.google.com/view/vitam-d/.

This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multilayer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively.

Tactile sensing is one of the modalities humans rely on heavily to perceive the world. Working with vision, this modality refines local geometry structure, measures defor-mation at the contact area, and indicates the hand-object contact state. With the availability of open-source tactile sensors such as DIGIT, research on visual-tactile learning is becoming more accessible and reproducible. Leveraging this tactile sensor, we propose a novel visual-tactile in-hand object reconstruction framework VTacO, and ex-tend it to VTacOH for hand-object reconstruction. Since our method can support both rigid and deformable ob-ject reconstruction, no existing benchmarks are proper for the goal. We propose a simulation environment, VT-Sim, which supports generating hand-object interaction for both rigid and deformable objects. With VT-Sim, we gener-ate a large-scale training dataset and evaluate our method on it. Extensive experiments demonstrate that our pro-posed method can outperform the previous baseline meth-ods qualitatively and quantitatively. Finally, we directly ap-ply our model trained in simulation to various real-world test cases, which display qualitative results. Codes, mod-els, simulation environment, and datasets are available at https://sites.google.com/view/vtaco/.

The accurate detection of clouds in images is prerequisite for remote sensing image processing and applications. Traditional cloud detection methods rely on particular sensors, and the artificial neural network method only uses spectral or spatial information. In this paper, a novel method combining multiple features based on deep learning (MDL) for cloud detection is proposed. Deep neural network (DNN) and fully convolutional neural (FCN) network are applied to extract the spectral and spatial features of the images respectively, and the features are used for the input of another DNN for re-learning while the image data also serves as an input to the DNN. Finally, joint feature obtained by relearning is classified by Support Vector Machine (SVM). The method makes full use of the spectral-spatial information of the images to detect cloud comprehensively. A comparative experiment was carried on Landsat 8 images containing different types of clouds over various underlying surfaces. The results show that the MDL method performs favorably, which is significantly improved compared to the single neural network algorithm and the function of mask (FMask) algorithm.