Deep understanding of human hand dexterity is crucial for making robotic hands more generalizable. While human hand manipulation skills, embedded in hand kinematics and tactile sensing, are typically recorded using instrumented gloves, these gloves can hinder natural hand movement and tactile feedback, potentially limiting the quality of recorded human manipulation data and adversely affecting the human manipulation understanding and the human–robot skill transfer process. We thus propose a novel approach for tactile inference by simultaneously capturing kinematic, electromyography, and tactile information during human manipulation without invasive data gloves. Autoencoder-transformer tactile network, a deep learning framework that leverages modality-specific autoencoders and a Transformer-based model, is introduced to extract compact latent representations from multiple modalities and accurately predict tactile information. We evaluated our approach using a dataset of human manipulation activities, where participants performed various tasks including frontal reaching for objects, pouring, screwing, and feeding, while their kinematics, electromyography, and tactile information were recorded. The proposed approach achieves a normalized root-mean-square error in tactile reconstruction of 0.032, a mean absolute error of 0.015, and a symmetric mean absolute percentage error of 13.4%, significantly outperforming standard baseline methods. These results demonstrated that our noninvasive approach could effectively infer tactile information while preserving natural hand movement and tactile feedback, leading to improved data quality that enhances both the understanding of human motor control and imitation learning for more nuanced and dexterous robotic control.

Wrist-worn step-counting holds potential to improve health management and disease prevention. However, inaccurate step counting is often caused by false positives during non-walking wrist movements, potentially leading to incorrect health assessments, ineffective interventions, and suboptimal patient outcomes. We thus propose a real-time adaptive multi-stage step counting algorithm based on a smartwatch 3-axis accelerometer, integrating non-walking detection to identify false positive step counts during non-walking wrist movements. Sixty-seven subjects wore a smartwatch with a 3-axis accelerometer and performed walking and running trials and eight non-gait trials: eating with forks and chopsticks, drinking, rolling while sleeping, flipping the wrist to check a watch, moving and grasping objects, typing, and using a computer mouse. When evaluated on the proprietary dataset, the proposed model was 93.58% accurate in estimating step counts as compared with 10.09% accuracy from a standard peak detection framework that grossly over-counted steps during non-walking movements. In non-walking detection experiments, the proposed model was almost more accurate and efficient than other four baseline models (p < 0.05), while requiring only 8.9% of the inference time of a single OCSVM. These results highlight the importance of rejecting false positive step counts during non-walking movements from wrist-worn step counters, and our proposed approach holds potential to more accurately estimate step counting in real-life scenarios to improve aerobic exercise assessment and promote sedentary disease prevention.

Rehabilitation training plays a vital role in the recovery of lower back and cervical spine function. Human pose estimation can support this process by guiding and evaluating rehabilitation movements. However, specialized rehabilitation exercises often involve severe self-occlusions, posing significant challenges for vision-based pose estimation methods. We thus propose a full-body pose estimation framework tailored for rehabilitation exercises, which fuses monocular images and inertial measurement unit (IMU) signals using a temporal transformer. Multimodal data was collected from six subjects performing 22 specialized rehabilitation movements (e.g., single-leg open book, cross-leg body rotation, standing iliotibial band stretch, standing lumbar extension). The collected data comprises synchronized images, 2D and 3D human keypoint coordinates, and IMU signals. Our approach first employs a convolutional neural network (CNN) to extract 2D keypoints from image sequences. These keypoints, combined with IMU signals, are then processed by a temporal transformer to estimate 3D joint coordinates. On the collected data, a vision-only baseline yields a 2D joint position error of${7.33} \pm {2.08}$pixels and a 3D joint error of${10.05} \pm {2.67}$cm. In comparison, the proposed method achieves lower errors, with${5.50} \pm {0.75}$pixels for 2D joints and$8.27 \pm 1.03 \text{cm}$for 3D joints. By leveraging inertial data, our method enhances the robustness of pose estimation under challenging conditions such as self-occlusion, demonstrating its potential for both clinical and home-based rehabilitation applications.

Electromyography (EMG) signals are widely used as control inputs for myoelectric exoskeletons. However, muscle fatigue, which can result from prolonged use or heavy loads, significantly affects muscle activation patterns, leading to reduced estimation accuracy. To address this challenge, we propose an adversarial learning framework to enhance grip force estimation under fatigue conditions. The framework consists of three key components: a domain-invariant feature extractor to mitigate domain shifts between non-fatigue and fatigue states, a force estimator to predict grip forces from these domain-invariant features, and a domain discriminator to distinguish between the two domains. The proposed method was evaluated on a dataset collected from eight participants performing gripping tasks under both non-fatigue and fatigue conditions, during which high-density EMG signals and grip forces were recorded simultaneously. Experimental results demonstrated that our method significantly reduced the root mean square error (RMSE) from 0.264 to 0.127, outperforming a baseline model consisting of only the feature extractor and force estimator$(p < 0.01)$. Additionally, the proposed approach exhibited consistent performance across all participants, highlighting its robustness and generalizability. These findings suggest that the proposed adversarial learning framework effectively enhances grip force estimation accuracy under muscle fatigue, offering a promising solution for improving the reliability and usability of myoelectric exoskeletons.