Deep understanding of human hand dexterity is crucial for making robotic hands more generalizable. While human hand manipulation skills, embedded in hand kinematics and tactile sensing, are typically recorded using instrumented gloves, these gloves can hinder natural hand movement and tactile feedback, potentially limiting the quality of recorded human manipulation data and adversely affecting the human manipulation understanding and the human–robot skill transfer process. We thus propose a novel approach for tactile inference by simultaneously capturing kinematic, electromyography, and tactile information during human manipulation without invasive data gloves. Autoencoder-transformer tactile network, a deep learning framework that leverages modality-specific autoencoders and a Transformer-based model, is introduced to extract compact latent representations from multiple modalities and accurately predict tactile information. We evaluated our approach using a dataset of human manipulation activities, where participants performed various tasks including frontal reaching for objects, pouring, screwing, and feeding, while their kinematics, electromyography, and tactile information were recorded. The proposed approach achieves a normalized root-mean-square error in tactile reconstruction of 0.032, a mean absolute error of 0.015, and a symmetric mean absolute percentage error of 13.4%, significantly outperforming standard baseline methods. These results demonstrated that our noninvasive approach could effectively infer tactile information while preserving natural hand movement and tactile feedback, leading to improved data quality that enhances both the understanding of human motor control and imitation learning for more nuanced and dexterous robotic control.

Wrist-worn step-counting holds potential to improve health management and disease prevention. However, inaccurate step counting is often caused by false positives during non-walking wrist movements, potentially leading to incorrect health assessments, ineffective interventions, and suboptimal patient outcomes. We thus propose a real-time adaptive multi-stage step counting algorithm based on a smartwatch 3-axis accelerometer, integrating non-walking detection to identify false positive step counts during non-walking wrist movements. Sixty-seven subjects wore a smartwatch with a 3-axis accelerometer and performed walking and running trials and eight non-gait trials: eating with forks and chopsticks, drinking, rolling while sleeping, flipping the wrist to check a watch, moving and grasping objects, typing, and using a computer mouse. When evaluated on the proprietary dataset, the proposed model was 93.58% accurate in estimating step counts as compared with 10.09% accuracy from a standard peak detection framework that grossly over-counted steps during non-walking movements. In non-walking detection experiments, the proposed model was almost more accurate and efficient than other four baseline models (p < 0.05), while requiring only 8.9% of the inference time of a single OCSVM. These results highlight the importance of rejecting false positive step counts during non-walking movements from wrist-worn step counters, and our proposed approach holds potential to more accurately estimate step counting in real-life scenarios to improve aerobic exercise assessment and promote sedentary disease prevention.

Rehabilitation training plays a vital role in the recovery of lower back and cervical spine function. Human pose estimation can support this process by guiding and evaluating rehabilitation movements. However, specialized rehabilitation exercises often involve severe self-occlusions, posing significant challenges for vision-based pose estimation methods. We thus propose a full-body pose estimation framework tailored for rehabilitation exercises, which fuses monocular images and inertial measurement unit (IMU) signals using a temporal transformer. Multimodal data was collected from six subjects performing 22 specialized rehabilitation movements (e.g., single-leg open book, cross-leg body rotation, standing iliotibial band stretch, standing lumbar extension). The collected data comprises synchronized images, 2D and 3D human keypoint coordinates, and IMU signals. Our approach first employs a convolutional neural network (CNN) to extract 2D keypoints from image sequences. These keypoints, combined with IMU signals, are then processed by a temporal transformer to estimate 3D joint coordinates. On the collected data, a vision-only baseline yields a 2D joint position error of${7.33} \pm {2.08}$pixels and a 3D joint error of${10.05} \pm {2.67}$cm. In comparison, the proposed method achieves lower errors, with${5.50} \pm {0.75}$pixels for 2D joints and$8.27 \pm 1.03 \text{cm}$for 3D joints. By leveraging inertial data, our method enhances the robustness of pose estimation under challenging conditions such as self-occlusion, demonstrating its potential for both clinical and home-based rehabilitation applications.

Wireless inertial motion capture holds promise for real-time human-machine interfaces and home-based rehabilitation applications. However, wireless data drop can cause significant estimation errors deteriorating performance or even making the system unusable. It is currently unclear how to estimate non-periodic kinematics with wearable inertial measurement units (IMUs) in the presence of wireless data drop (packet loss). We thus propose a novel inference encoder-decoder network model for real-time kinematics during dynamic movement. Twenty-four healthy subjects performed yoga, golf, swimming, dance, and badminton movement activities while wearing IMUs and 10-90% of each IMU's data were randomly removed to determine the effects of data drop on estimation accuracy with and without the proposed model. Results demonstrated a reduction in RMSE of 45.2% to 51.5% in the upper limb kinematic estimation of the proposed model compared to the No Prediction strategy, and a reduction of 19.1% to 31.3% of the proposed model compared with an baseline LSTM model. In addition, the proposed model has significantly less error (p<0.05) than the No Prediction strategy and the baseline LSTM model for 10%, 20%, 30%, 40%, 50%, 60%, 70%, and 80% data drop. These results could enable wearable, wireless IMU dynamic motion analysis and assessment with reduced kinematic estimation error in the presence of varying amounts of wireless data drop and thus could further facilitate human-machine interaction and home-based medical assessment and treatment.

Estimating hand and body posture is crucial for enabling human-robot collaboration, preventing occupational diseases, and training humanoid robots. Although advances in wearable motion sensors, such as Inertial Measurement Units (IMUs), have resulted in public datasets in industrial and occupational settings, these datasets rarely include movements with subjects holding and manipulating objects. However, it is crucial to have data on how humans move and interact with different objects so that we can better understand human motion intention and movement strategies in specific scenarios. We thus propose the HBOD dataset (hand-body-object dataset), which encompasses synchronized human pose data from an IMU sensor network, hand posture data from a smart data glove, and object position and attitude information obtained from a motion capture system, while subjects move and interact with a screwdriver, hammer, spanner, electric drill, and a rectangular workpiece. This paper provides an overview of the hardware setup, experimental protocol, data format, and data visualization results. This dataset provides crucial object information absent from existing datasets, thus offering highly valuable manipulation data for occupational diseases research, human-robot interaction, and robot skill acquisition.