Deep understanding of human hand dexterity is crucial for making robotic hands more generalizable. While human hand manipulation skills, embedded in hand kinematics and tactile sensing, are typically recorded using instrumented gloves, these gloves can hinder natural hand movement and tactile feedback, potentially limiting the quality of recorded human manipulation data and adversely affecting the human manipulation understanding and the human–robot skill transfer process. We thus propose a novel approach for tactile inference by simultaneously capturing kinematic, electromyography, and tactile information during human manipulation without invasive data gloves. Autoencoder-transformer tactile network, a deep learning framework that leverages modality-specific autoencoders and a Transformer-based model, is introduced to extract compact latent representations from multiple modalities and accurately predict tactile information. We evaluated our approach using a dataset of human manipulation activities, where participants performed various tasks including frontal reaching for objects, pouring, screwing, and feeding, while their kinematics, electromyography, and tactile information were recorded. The proposed approach achieves a normalized root-mean-square error in tactile reconstruction of 0.032, a mean absolute error of 0.015, and a symmetric mean absolute percentage error of 13.4%, significantly outperforming standard baseline methods. These results demonstrated that our noninvasive approach could effectively infer tactile information while preserving natural hand movement and tactile feedback, leading to improved data quality that enhances both the understanding of human motor control and imitation learning for more nuanced and dexterous robotic control.

Understanding the underlying mechanism of neuromuscular system on motion/force generation is essential for human-machine interfacing. However, simultaneous decoding of wrist angles and grasp forces from neural signals remains an open challenge in the field of neural interfacing. In this study, we proposed a scheme leveraging channel-wise cumulative spike trains (cw-CSTs) of motor units to simultaneously decode wrist angles and grasp forces. Specifically, a spatial spike detection method was utilized to detect cw-CST from surface electromyography, observing as much as possible of motor unit activities. Accordingly, we extracted three neural features to drive the decoders, including a twitch force model-based (cw-MUdrive) and a discharge rate-based (DR-cwCST) neural features derived from cw-CSTs, and DR of motor units (DR-MUST) decomposed by a conventional blind source separation algorithm. Wrist- and hand-specific decoders were built to estimate wrist angles and grasp forces via Gaussian process regression. Experiments were conducted with ten subjects, in which they activated wrist motions and grasp forces concurrently. We evaluated the performance with both accuracy and output stability. Results demonstrated that the cwCST-based neural features outperformed the conventional DR-MUST features with both higher accuracy and stability metrics. Additionally, cw-MUdrive performed better than DR-cwCST in grasp force estimation and comparable to DR-cwCST in wrist angle estimation. The outcome provides an effective solution for simultaneously decoding wrist movements and hand grasp forces, promoting the development of natural control in neural interface.

Electromyography (EMG) signals are widely used as control inputs for myoelectric exoskeletons. However, muscle fatigue, which can result from prolonged use or heavy loads, significantly affects muscle activation patterns, leading to reduced estimation accuracy. To address this challenge, we propose an adversarial learning framework to enhance grip force estimation under fatigue conditions. The framework consists of three key components: a domain-invariant feature extractor to mitigate domain shifts between non-fatigue and fatigue states, a force estimator to predict grip forces from these domain-invariant features, and a domain discriminator to distinguish between the two domains. The proposed method was evaluated on a dataset collected from eight participants performing gripping tasks under both non-fatigue and fatigue conditions, during which high-density EMG signals and grip forces were recorded simultaneously. Experimental results demonstrated that our method significantly reduced the root mean square error (RMSE) from 0.264 to 0.127, outperforming a baseline model consisting of only the feature extractor and force estimator$(p < 0.01)$. Additionally, the proposed approach exhibited consistent performance across all participants, highlighting its robustness and generalizability. These findings suggest that the proposed adversarial learning framework effectively enhances grip force estimation accuracy under muscle fatigue, offering a promising solution for improving the reliability and usability of myoelectric exoskeletons.

Ensuring robust tracking of controllers’ movement is critical for human-robot interaction in virtual reality (VR) scenarios. This paper proposes a robust tracking algorithm based on a novel wearable ring-shaped controller equipped with an inertial measurement unit (IMU) and a light-emitting diode (LED). This novel controller design allows users to free up their hands for more immersive experiences. To track the controller’s motion accurately and robustly, we resort to various forms of visual measurements, including 6 DoF and 5 DoF pose measurements from hand gesture detection, as well as 3 DoF position measurement and 2 DoF image measurement derived from the LED. We theoretically analyze the performances of these observation models and propose an optimal observation model combination scheme. Moreover, the necessity and rationale of online estimating system gravity are illustrated. The effectiveness of our tracking method is validated through extensive experiments.

Myoelectric prostheses are generally unable to accurately control the position and force simultaneously, prohibiting natural and intuitive human-machine interaction. This issue is attributed to the limitations of myoelectric interfaces in effectively decoding multi-degree-of-freedom (multi-DoF) kinematic and kinetic information. We thus propose a novel multi-task, spatial-temporal model driven by graphical high-density electromyography (HD-EMG) for simultaneous and proportional control of wrist angle and grasp force. Twelve subjects were recruited to perform three multi-DoF movements, including wrist pronation/supination, wrist flexion/extension, and wrist abduction/adduction while varying grasp force. Experimental results demonstrated that the proposed model outperformed five baseline models, with the normalized root mean square error of 13.2% and 9.7% and the correlation coefficient of 89.6% and 91.9% for wrist angle and grasp force estimation, respectively. In addition, the proposed model still maintained comparable accuracy even with a significant reduction in the number of HD-EMG electrodes. To the best of our knowledge, this is the first study to achieve simultaneous and proportional wrist angle and grasp force control via HD-EMG and has the potential to empower prostheses users to perform a broader range of tasks with greater precision and control, ultimately enhancing their independence and quality of life.

Wireless inertial motion capture holds promise for real-time human-machine interfaces and home-based rehabilitation applications. However, wireless data drop can cause significant estimation errors deteriorating performance or even making the system unusable. It is currently unclear how to estimate non-periodic kinematics with wearable inertial measurement units (IMUs) in the presence of wireless data drop (packet loss). We thus propose a novel inference encoder-decoder network model for real-time kinematics during dynamic movement. Twenty-four healthy subjects performed yoga, golf, swimming, dance, and badminton movement activities while wearing IMUs and 10-90% of each IMU's data were randomly removed to determine the effects of data drop on estimation accuracy with and without the proposed model. Results demonstrated a reduction in RMSE of 45.2% to 51.5% in the upper limb kinematic estimation of the proposed model compared to the No Prediction strategy, and a reduction of 19.1% to 31.3% of the proposed model compared with an baseline LSTM model. In addition, the proposed model has significantly less error (p<0.05) than the No Prediction strategy and the baseline LSTM model for 10%, 20%, 30%, 40%, 50%, 60%, 70%, and 80% data drop. These results could enable wearable, wireless IMU dynamic motion analysis and assessment with reduced kinematic estimation error in the presence of varying amounts of wireless data drop and thus could further facilitate human-machine interaction and home-based medical assessment and treatment.

Assessing ankle joint power during real-life scenarios is crucial for analyzing human push-off and detecting abnormal gait patterns. However, traditional joint power monitoring methods require expensive and professional equipment, limiting their use to gait laboratories. To address this limitation, we propose a portable and robust two-stage approach that estimates ankle joint power using two inertial measurement units (IMU) sensors placed on the shank and foot, respectively. Our subject-independent CNN model accurately assessed ankle joint power during flat and inclined walking across 28 walking speeds and 6 ramp inclines. This solution facilitates ankle joint power assessment outside of gait laboratories and could serve as a foundation to enable gait abnormality evaluation in patients in hospitals, clinics, and home-based settings.

This work investigates real-time estimation of vertical ground reaction force (vGRF) and external knee extension moment (KEM) during single- and double-leg drop landings via wearable inertial measurement units (IMUs) and machine learning. A real-time, modular LSTM model with four sub-deep neural networks was developed to estimate vGRF and KEM. Sixteen subjects wore eight IMUs on the chest, waist, right and left thighs, shanks, and feet and performed drop landing trials. Ground embedded force plates and an optical motion capture system were used for model training and evaluation. During single-leg drop landings, accuracy for the vGRF and KEM estimation was$R^{2}$= 0.88$\pm$0.12 and$R^{2}$= 0.84$\pm$0.14, respectively, and during double-leg drop landings, accuracy for the vGRF and KEM estimation was$R^{2}$= 0.85$\pm$0.11 and$R^{2}$= 0.84$\pm$0.12, respectively. The best vGRF and KEM estimations of the model with the optimal LSTM unit number (130) require eight IMUs placed on the eight selected locations during single-leg drop landings. During double-leg drop landings, the best estimation on a leg only needs five IMUs placed on the chest, waist, and the leg's shank, thigh, and foot. The proposed modular LSTM-based model with optimally-configurable wearable IMUs can accurately estimate vGRF and KEM in real-time with relatively low computational cost during single- and double-leg drop landing tasks. This investigation could potentially enable in-field, non-contact anterior cruciate ligament injury risk screening and intervention training programs.

Autonomous mobile vehicles are expected to perform persistent and accurate localization with low-cost equipment. To achieve this goal, we propose a stereo camera based visual localization method using a modified laser map, which takes the advantage of both the low cost of camera, and high geometric precision of laser data to achieve long-term performance. Considering that LiDAR and camera give measurements of the same environment in different modalities, the cross-modal invariance is investigated to modify the laser map for visual localization. Specifically, a map learning algorithm is introduced to sample the robust subsets in laser maps that are useful for visual localization using multi-session visual and laser data. Further, a generative map model is derived to describe this cross-modal invariance, based on which two types of measurements are defined to model the laser map points as appropriate visual observations. Tightly coupling these measurements within the local bundle adjustment during online sliding-window based visual odometry, the vehicle can achieve robust localization even one year after the map was built. The effectiveness of the proposed method is evaluated on both the public KITTI datasets and self-collected datasets in our campus, which include seasonal, illumination and object variations. On all experimental localization sessions, our method provides satisfactory results, even when the direction is opposite to that in the mapping session, verifying the superior performance of the laser map based visual localization method.

Long-term visual localization in outdoor environment is a challenging problem, especially faced with the cross-seasonal, bi-directional tasks and changing environment. In this paper we propose a novel visual inertial localization framework that localizes against the LiDAR-built map. Based on the geometry information of the laser map, a hybrid bundle adjustment framework is proposed, which estimates the poses of the cameras with respect to the prior laser map as well as optimizes the state variables of the online visual inertial odometry system simultaneously. For more accurate crossmodal data association, the laser map is optimized using multisession laser and visual data to extract the salient and stable subset for visual localization. To validate the efficiency of the proposed method, we collect data in south part of our campus in different seasons, along the same and opposite-direction route. In all sessions of localization data, our proposed method gives satisfactory results, and shows the superiority of the hybrid bundle adjustment and map optimization1.