Deep understanding of human hand dexterity is crucial for making robotic hands more generalizable. While human hand manipulation skills, embedded in hand kinematics and tactile sensing, are typically recorded using instrumented gloves, these gloves can hinder natural hand movement and tactile feedback, potentially limiting the quality of recorded human manipulation data and adversely affecting the human manipulation understanding and the human–robot skill transfer process. We thus propose a novel approach for tactile inference by simultaneously capturing kinematic, electromyography, and tactile information during human manipulation without invasive data gloves. Autoencoder-transformer tactile network, a deep learning framework that leverages modality-specific autoencoders and a Transformer-based model, is introduced to extract compact latent representations from multiple modalities and accurately predict tactile information. We evaluated our approach using a dataset of human manipulation activities, where participants performed various tasks including frontal reaching for objects, pouring, screwing, and feeding, while their kinematics, electromyography, and tactile information were recorded. The proposed approach achieves a normalized root-mean-square error in tactile reconstruction of 0.032, a mean absolute error of 0.015, and a symmetric mean absolute percentage error of 13.4%, significantly outperforming standard baseline methods. These results demonstrated that our noninvasive approach could effectively infer tactile information while preserving natural hand movement and tactile feedback, leading to improved data quality that enhances both the understanding of human motor control and imitation learning for more nuanced and dexterous robotic control.

A major issue after total knee replacement (TKR) surgery is asymmetric gait kinetics, which increases knee loads on the non-operated knee. This imbalance accelerates osteoarthritis (OA) progression, often leading to a second contralateral TKR. There is a clear need for an advanced wearable system with multiple sensors to accurately estimate gait kinetics in natural environments. This study aims to develop a machine learning framework that exclusively uses wearable inertial measurement units (IMUs) during overground and treadmill walking to estimate knee flexion moment (KFM) and knee adduction moment (KAM), significant biomechanical factors linked to OA. We introduce a novel deep learning model that combines a Long Short-Term Memory (LSTM)-based Autoencoder and Variational Gaussian Process (VGP) to estimate the mean and uncertainty region of the KAM and KFM. Seventeen healthy participants performed treadmill walking trials, while a separate group of seventeen healthy participants performed overground walking trials for model training and validation. Results demonstrated Root Mean Square Errors (RMSE) of 0.49%BW$\cdot$BH (body weight × body height) and 0.73%BW$\cdot$BH for KAM and KFM, respectively, during treadmill walking and 0.74%BW$\cdot$BH and 0.49%BW$\cdot$BH for KAM and KFM respectively during overground walking, which is more accurate than existing approaches. The proposed model with wearable IMUs could enable knee health monitoring and rehabilitation for these key biomechanical factors linked to the progression of knee joint pathologies outside of traditional biomechanical laboratories with large, tethered equipment and into clinics, hospitals, and the community.

Wrist-worn step-counting holds potential to improve health management and disease prevention. However, inaccurate step counting is often caused by false positives during non-walking wrist movements, potentially leading to incorrect health assessments, ineffective interventions, and suboptimal patient outcomes. We thus propose a real-time adaptive multi-stage step counting algorithm based on a smartwatch 3-axis accelerometer, integrating non-walking detection to identify false positive step counts during non-walking wrist movements. Sixty-seven subjects wore a smartwatch with a 3-axis accelerometer and performed walking and running trials and eight non-gait trials: eating with forks and chopsticks, drinking, rolling while sleeping, flipping the wrist to check a watch, moving and grasping objects, typing, and using a computer mouse. When evaluated on the proprietary dataset, the proposed model was 93.58% accurate in estimating step counts as compared with 10.09% accuracy from a standard peak detection framework that grossly over-counted steps during non-walking movements. In non-walking detection experiments, the proposed model was almost more accurate and efficient than other four baseline models (p < 0.05), while requiring only 8.9% of the inference time of a single OCSVM. These results highlight the importance of rejecting false positive step counts during non-walking movements from wrist-worn step counters, and our proposed approach holds potential to more accurately estimate step counting in real-life scenarios to improve aerobic exercise assessment and promote sedentary disease prevention.

Rehabilitation training plays a vital role in the recovery of lower back and cervical spine function. Human pose estimation can support this process by guiding and evaluating rehabilitation movements. However, specialized rehabilitation exercises often involve severe self-occlusions, posing significant challenges for vision-based pose estimation methods. We thus propose a full-body pose estimation framework tailored for rehabilitation exercises, which fuses monocular images and inertial measurement unit (IMU) signals using a temporal transformer. Multimodal data was collected from six subjects performing 22 specialized rehabilitation movements (e.g., single-leg open book, cross-leg body rotation, standing iliotibial band stretch, standing lumbar extension). The collected data comprises synchronized images, 2D and 3D human keypoint coordinates, and IMU signals. Our approach first employs a convolutional neural network (CNN) to extract 2D keypoints from image sequences. These keypoints, combined with IMU signals, are then processed by a temporal transformer to estimate 3D joint coordinates. On the collected data, a vision-only baseline yields a 2D joint position error of${7.33} \pm {2.08}$pixels and a 3D joint error of${10.05} \pm {2.67}$cm. In comparison, the proposed method achieves lower errors, with${5.50} \pm {0.75}$pixels for 2D joints and$8.27 \pm 1.03 \text{cm}$for 3D joints. By leveraging inertial data, our method enhances the robustness of pose estimation under challenging conditions such as self-occlusion, demonstrating its potential for both clinical and home-based rehabilitation applications.

Electromyography (EMG) signals are widely used as control inputs for myoelectric exoskeletons. However, muscle fatigue, which can result from prolonged use or heavy loads, significantly affects muscle activation patterns, leading to reduced estimation accuracy. To address this challenge, we propose an adversarial learning framework to enhance grip force estimation under fatigue conditions. The framework consists of three key components: a domain-invariant feature extractor to mitigate domain shifts between non-fatigue and fatigue states, a force estimator to predict grip forces from these domain-invariant features, and a domain discriminator to distinguish between the two domains. The proposed method was evaluated on a dataset collected from eight participants performing gripping tasks under both non-fatigue and fatigue conditions, during which high-density EMG signals and grip forces were recorded simultaneously. Experimental results demonstrated that our method significantly reduced the root mean square error (RMSE) from 0.264 to 0.127, outperforming a baseline model consisting of only the feature extractor and force estimator$(p < 0.01)$. Additionally, the proposed approach exhibited consistent performance across all participants, highlighting its robustness and generalizability. These findings suggest that the proposed adversarial learning framework effectively enhances grip force estimation accuracy under muscle fatigue, offering a promising solution for improving the reliability and usability of myoelectric exoskeletons.

Magnetic soft robots have the potential to be used in biomedical applications, such as targeted drug delivery, minimally invasive surgery, and on-chip tissue manipulation, due to their untethered operation, rapid actuation, and physical adaptability. However, real-time control of these robots is challenging because of their inherent nonlinear dynamics, fabrication imperfections, and complex interactions with external magnetic fields. In this work, we present a model-free controller that uses Proximal Policy Optimization-collected experience and a hybrid architecture, EpisodicMemNet. A memory module returns stored actions for angle-matched states, while a multi-head network predicts actions when no match is found. In experimental validation on two magnetic soft robots, a four-legged dual-stem H-frame (6 × 10 mm) and a three-legged asymmetric variant (8 × 11 mm), an adaptive two-tier memory maintained a median lookup time of approximately 1.2 ms, and EpisodicMemNet achieved 87.4% balanced accuracy and 93.5% top-2 accuracy while sustaining 2-Hz real-time control. Furthermore, user-guided tasks such as target navigation, obstacle avoidance, and ramp climbing confirmed reliable performance and adaptability despite the system’s nonlinearities and data sparsity. The proposed method thus not only overcomes the limitations of traditional simulation-based and model-specific approaches but also paves the way for scalable, experience-driven control solutions in soft robotic applications.

Step width is vital for gait stability, postural balance control, and fall risk reduction. However, estimating step width typically requires either fixed cameras or a full kinematic body suit of wearable inertial measurement units (IMUs), both of which are often too expensive and time-consuming for clinical application. We thus propose a novel data-augmented deep learning model for estimating step width in individuals with and without neurodegenerative disease using a minimal set of wearable IMUs. Twelve patients with neurodegenerative, clinically diagnosed Spinocerebellar ataxia type 3 (SCA3) performed over ground walking trials, and seventeen healthy individuals performed treadmill walking trials at various speeds and gait modifications while wearing IMUs on each shank and the pelvis. Results demonstrated step width mean absolute errors of 3.3 0.7 cm and 2.9 0.5 cm for the neurodegenerative and healthy groups, respectively, which were below the minimal clinically important difference of 6.0 cm. Step width variability mean absolute errors were 1.5 cm and 0.8 cm for neurodegenerative and healthy groups, respectively. Data augmentation significantly improved accuracy performance in the neurodegenerative group, likely because they exhibited larger variations in walking kinematics as compared with healthy subjects. These results could enable clinically meaningful and accurate portable step width monitoring for individuals with and without neurodegenerative disease, potentially enhancing rehabilitative training, assessment, and dynamic balance control in clinical and real-life settings.

Human activity recognition (HAR) enables real-time monitoring of human movement, posture, and activity level, and can provide valuable information for health management. With the continuous advancement of Internet of Things (IoT) technology, wearable sensors and smartphones equipped with various types of sensors have become widely utilized to collect multimodal data for HAR. However, in multimodal HAR, current fusion methods fall short in capturing inter-modality correlations, hampering the full exploitation of complementary information between modalities and leading to lower recognition accuracy. We thus propose a novel multiscale cross-modal interactive fusion network (MCIFN), which can fully capture correlations between various modalities and obtain an effective fused representation for HAR. Specifically, we employ a multiscale parallel convolution module to extract features from each modality at multiple scales. Then, an interactive fusion strategy based on the cross-modal attention mechanism is introduced to adjust and enhance each modality based on its correlations with other modalities. Additionally, to resolve the information redundancy caused by the interactive fusion strategy, we utilize a hybrid attention module to focus on important information in the fusion representation. Extensive experiments conducted on three publicly available datasets and one private dataset demonstrate that our proposed network outperforms the previous baseline networks for HAR. Additionally, our proposed fusion strategy yielded a notable improvement in accuracy ranging from 1.87% to 9.96% compared to existing strategies. These findings imply that our newly proposed network can realize comprehensive multimodal fusion and effectively enhance HAR accuracy, potentially contributing to advancements in individual health management and personalized healthcare interventions.

Myoelectric prostheses are generally unable to accurately control the position and force simultaneously, prohibiting natural and intuitive human-machine interaction. This issue is attributed to the limitations of myoelectric interfaces in effectively decoding multi-degree-of-freedom (multi-DoF) kinematic and kinetic information. We thus propose a novel multi-task, spatial-temporal model driven by graphical high-density electromyography (HD-EMG) for simultaneous and proportional control of wrist angle and grasp force. Twelve subjects were recruited to perform three multi-DoF movements, including wrist pronation/supination, wrist flexion/extension, and wrist abduction/adduction while varying grasp force. Experimental results demonstrated that the proposed model outperformed five baseline models, with the normalized root mean square error of 13.2% and 9.7% and the correlation coefficient of 89.6% and 91.9% for wrist angle and grasp force estimation, respectively. In addition, the proposed model still maintained comparable accuracy even with a significant reduction in the number of HD-EMG electrodes. To the best of our knowledge, this is the first study to achieve simultaneous and proportional wrist angle and grasp force control via HD-EMG and has the potential to empower prostheses users to perform a broader range of tasks with greater precision and control, ultimately enhancing their independence and quality of life.

Wireless inertial motion capture holds promise for real-time human-machine interfaces and home-based rehabilitation applications. However, wireless data drop can cause significant estimation errors deteriorating performance or even making the system unusable. It is currently unclear how to estimate non-periodic kinematics with wearable inertial measurement units (IMUs) in the presence of wireless data drop (packet loss). We thus propose a novel inference encoder-decoder network model for real-time kinematics during dynamic movement. Twenty-four healthy subjects performed yoga, golf, swimming, dance, and badminton movement activities while wearing IMUs and 10-90% of each IMU's data were randomly removed to determine the effects of data drop on estimation accuracy with and without the proposed model. Results demonstrated a reduction in RMSE of 45.2% to 51.5% in the upper limb kinematic estimation of the proposed model compared to the No Prediction strategy, and a reduction of 19.1% to 31.3% of the proposed model compared with an baseline LSTM model. In addition, the proposed model has significantly less error (p<0.05) than the No Prediction strategy and the baseline LSTM model for 10%, 20%, 30%, 40%, 50%, 60%, 70%, and 80% data drop. These results could enable wearable, wireless IMU dynamic motion analysis and assessment with reduced kinematic estimation error in the presence of varying amounts of wireless data drop and thus could further facilitate human-machine interaction and home-based medical assessment and treatment.

Estimating hand and body posture is crucial for enabling human-robot collaboration, preventing occupational diseases, and training humanoid robots. Although advances in wearable motion sensors, such as Inertial Measurement Units (IMUs), have resulted in public datasets in industrial and occupational settings, these datasets rarely include movements with subjects holding and manipulating objects. However, it is crucial to have data on how humans move and interact with different objects so that we can better understand human motion intention and movement strategies in specific scenarios. We thus propose the HBOD dataset (hand-body-object dataset), which encompasses synchronized human pose data from an IMU sensor network, hand posture data from a smart data glove, and object position and attitude information obtained from a motion capture system, while subjects move and interact with a screwdriver, hammer, spanner, electric drill, and a rectangular workpiece. This paper provides an overview of the hardware setup, experimental protocol, data format, and data visualization results. This dataset provides crucial object information absent from existing datasets, thus offering highly valuable manipulation data for occupational diseases research, human-robot interaction, and robot skill acquisition.

Assessing ankle joint power during real-life scenarios is crucial for analyzing human push-off and detecting abnormal gait patterns. However, traditional joint power monitoring methods require expensive and professional equipment, limiting their use to gait laboratories. To address this limitation, we propose a portable and robust two-stage approach that estimates ankle joint power using two inertial measurement units (IMU) sensors placed on the shank and foot, respectively. Our subject-independent CNN model accurately assessed ankle joint power during flat and inclined walking across 28 walking speeds and 6 ramp inclines. This solution facilitates ankle joint power assessment outside of gait laboratories and could serve as a foundation to enable gait abnormality evaluation in patients in hospitals, clinics, and home-based settings.

Hyperbolic localization measures the time difference of arrivals (TDOAs) of signals to determine the location of a wireless source or receiver. Traditional methods depend on precise clock synchronization between nodes so that time measurements from independent devices can be meaningfully compared. Imperfect synchronization is often the dominant source of error. We propose two new message-based TDOA equations for hyperbolic localization which require no synchronization and meet or exceed state-of-the-art accuracy. Our approaches leverage anchor nodes that observe each other’s packet arrival times and a novel reformulation of the TDOA equation to reduce the effect of clock drift error. Closed-form equations are derived for computing TDOA in both self-localization and source-localization modes of operation along with bounds on maximum clock drift error. Three experiments are performed, including a clock drift simulation, a nonline-of-sight (NLOS) simulation, and an indoor validation experiment on custom ultra wideband (UWB) hardware all of which involved eight anchor nodes and one localizing node in a 128-$\text{m}^{3}$capture volume. Our source-localization approach achieved unprecedented accuracy with lower cost equipment and trivial setup. Our self-localization matched state-of-the-art accuracy but with infinite scalability and high privacy. These results could enable economical and infinite density indoor navigation and dramatically reduce the economic cost and increase the accuracy of implementing industrial and commercial tracking applications.

Wearable sensing and computer vision could move biomechanics from specialized laboratories to natural environments, but better algorithms are needed to extract meaningful outcomes from these emerging modalities. In this article, we present new models for estimating biomechanical outcomes—the knee adduction moment (KAM) and knee flexion moment (KFM)—from fusion of smartphone cameras and wearable inertial measurement units (IMUs) among young healthy nonobese males. A deep learning model was developed to extract features, fuse multimodal data, and estimate KAM and KFM. Walking data from 17 subjects were recorded with eight IMUs and two smartphone cameras. The model that used IMU-camera fusion was significantly more accurate than those using IMUs or cameras alone. The root-mean-square errors of the fusion model were 0.49$\%\;\mathbf {BW}\cdot \mathbf {BH}$for KAM and 0.66$\%\;\mathbf {BW}\cdot \mathbf {BH}$for KFM estimation, which are lower than clinically significant thresholds. With larger and more diverse data, this model could enable assessment of knee moments in clinics and homes.

This work investigates real-time estimation of vertical ground reaction force (vGRF) and external knee extension moment (KEM) during single- and double-leg drop landings via wearable inertial measurement units (IMUs) and machine learning. A real-time, modular LSTM model with four sub-deep neural networks was developed to estimate vGRF and KEM. Sixteen subjects wore eight IMUs on the chest, waist, right and left thighs, shanks, and feet and performed drop landing trials. Ground embedded force plates and an optical motion capture system were used for model training and evaluation. During single-leg drop landings, accuracy for the vGRF and KEM estimation was$R^{2}$= 0.88$\pm$0.12 and$R^{2}$= 0.84$\pm$0.14, respectively, and during double-leg drop landings, accuracy for the vGRF and KEM estimation was$R^{2}$= 0.85$\pm$0.11 and$R^{2}$= 0.84$\pm$0.12, respectively. The best vGRF and KEM estimations of the model with the optimal LSTM unit number (130) require eight IMUs placed on the eight selected locations during single-leg drop landings. During double-leg drop landings, the best estimation on a leg only needs five IMUs placed on the chest, waist, and the leg's shank, thigh, and foot. The proposed modular LSTM-based model with optimally-configurable wearable IMUs can accurately estimate vGRF and KEM in real-time with relatively low computational cost during single- and double-leg drop landing tasks. This investigation could potentially enable in-field, non-contact anterior cruciate ligament injury risk screening and intervention training programs.

Road surface condition monitoring and bike riding event detection are crucial in densely populated cities for travel efficiency and rider safety. However, most current approaches are either costly, unreliable in different scenarios, or not adaptable in new environments. This article proposes a novel automated approach leveraging widely used shared bikes to intelligently detect road surface conditions and riding events suitable for interactive Internet of Things (IoT) cities. We propose a novel dual stream meta learning approach to solve the reliability problem when bike types for the training and testing are different with a limited set of new samples and the self-adaptive problem when classifying new classes without retraining the model, both via dual stream meta learning. Results demonstrate the feasibility of the proposed IoT-based solution with 98.9% accuracy for road surface conditions and 99.6% accuracy for riding events via the proposed dual stream deep learning method in the conventional scenario. With few samples per class, the proposed method is more reliable than other commonly used approaches in the different-bike scenario (e.g., proposed 92.4% versus random forest 74.6%). In cases of predicting new classes, the algorithm is 95.6% accurate using only one sample per class without explicit training (compared to 78.0% for$K $-nearest neighbor). This article proposes a robust IoT framework for smart cities involving road surface conditions and rider events which could be critical for many applications, including city mapping, shared bike rental maintenance and rider performance, and city maintenance services.

Knee osteoarthritis is a progressive disease mediated by high joint loads. Foot progression angle modifications that reduce the knee adduction moment (KAM), a surrogate of knee loading, have demonstrated efficacy in alleviating pain and improving function. Although changes to the foot progression angle are overall beneficial, KAM reductions are not consistent across patients. Moreover, customized interventions are time-consuming and require instrumentation not commonly available in the clinic. We present a regression model that uses minimal clinical data-a set of six features easily obtained in the clinic-to predict the extent of first peak KAM reduction after toe-in gait retraining. For such a model to generalize, the training data must be large and variable. Given the lack of large public datasets that contain different gaits for the same patient, we generated this dataset synthetically. Insights learned from a ground-truth dataset with both baseline and toe-in gait trials (N = 12) enabled the creation of a large (N = 138) synthetic dataset for training the predictive model. On a test set of data collected by a separate research group (N = 15), the first peak KAM reduction was predicted with a mean absolute error of 0.134% body weight * height (%BW*HT). This error is smaller than the standard deviation of the first peak KAM during baseline walking averaged across test subjects (0.306%BW*HT). This work demonstrates the feasibility of training predictive models with synthetic data and provides clinicians with a new tool to predict the outcome of patient-specific gait retraining without requiring gait lab instrumentation.

Non-line-of-sight (NLoS) conditions are a known performance limiting factor in ultrawideband (UWB) ranging accuracy, particularly in environments with densely packed dynamically moving humans, i.e., crowds. As UWB technology is recently seeing wide deployment in consumer devices, the effects of NLoS conditions due to human body shadowing will greatly degrade the performance of related ranging and localization applications. We thus propose the round-robin ranging protocol, an extension of alternative double-sided two-way ranging, which scales well for high rate sampling of dense networks and further propose a cooperative statistical approach based on multidimensional scaling (MDS) to mitigate NLoS conditions. Experimental validation was performed in a densely packed dynamically moving human environment. Nine subjects walked naturally within a 16-m2motion capture area while wearing UWB sensors at the foot, wrist, torso, and head and reflective optical motion capture markers. UWB accuracy of the 36 intersubject ranges was computed as the difference in range estimates between the UWB sensors and the optical motion capture system. Results showed that the proposed MDS approach reduced ranging accuracy errors by 11%, 13%, 16%, and 32% at the foot, wrist, torso, and head, respectively. Body placement had a significant effect on the ranging performance in that transceivers placed on the head and foot reduced ranging errors by 84% and 63% compared with the traditional torso placement. These results suggest that the proposed approaches could significantly improve the performance in UWB ranging applications involving human bodies such as human movement monitoring, athletics, logistics, and social contact tracing applications.

Walking, one of the most common daily activities, causes unwanted movement artifacts which can significantly deteriorate hand gesture recognition accuracy. However, traditional hand gesture recognition algorithms are typically developed and validated with wrist-worn devices only during static human poses, neglecting the critical importance of dynamic effects on gesture accuracy. Thus, we developed and validated a signal decomposition approach via empirical mode decomposition to accurately segment target gestures from coupled raw signals during dynamic walking and a transfer learning method based on distribution adaptation to enable gesture recognition through domain transfer between dynamic walking and static standing scenarios. Ten healthy subjects performed seven hand gestures during both walking and standing experiments while wearing an IMU wrist-worn device. Experimental results showed that the signal decomposition approach reduced the gesture detection error by 83.8%, and the transfer learning approach (20% transfer rate) improved hand gesture recognition accuracy by 15.1%. This ground-breaking work demonstrates the feasibility of hand gesture recognition while walking via wrist-worn sensing. These findings serve to inform real-life and ubiquitous adoption of wrist-worn hand gesture recognition for intuitive human-machine interaction in dynamic walking situations.

Wearable activity recognition can collate the type, intensity, and duration of each child's physical activity profile, which is important for exploring underlying adolescent health mechanisms. Traditional machine-learning-based approaches require large labeled data sets; however, child activity data sets are typically small and insufficient. Thus, we proposed a transfer learning approach that adapts adult-domain data to train a high-fidelity, subject-independent model for child activity recognition. Twenty children and twenty adults wore an accelerometer wristband while performing walking, running, sitting, and rope skipping activities. Activity classification accuracy was determined via the traditional machine learning approach without transfer learning and with the proposed subject-independent transfer learning approach. Results showed that transfer learning increased classification accuracy to 91.4% as compared to 80.6% without transfer learning. These results suggest that subject-independent transfer learning can improve accuracy and potentially reduce the size of the required child data sets to enable physical activity monitoring systems to be adopted more widely, quickly, and economically for children and provide deeper insights into injury prevention and health promotion strategies.

Wrist-based hand gesture recognition has the potential to unlock naturalistic human-computer interaction for a vast array of virtual and augmented reality applications. Photoplethysmography (PPG), force myography (FMG), and accelerometry (ACC) have generally been proposed as isolated single sensing modalities for gesture recognition, but any of these alone is inherently limited in the amount of biological information it can collect during finger and hand movements. We thus propose a novel, wrist-based, PPG-FMG-ACC combined sensing approach based on a multi-head attention mechanism fusion convolutional neural network (CNN-AF) for gesture recognition. Nine subjects performed twelve hand gestures involving various wrist and finger postures. Experimental results showed that multi-modal fusion improved classification performance significantly ( p 0.01) compared to any single sensing modality, and the F1-score of the combined PPG-FMG-ACC approach was 40.1% higher than PPG alone, 27.4% higher than ACC alone, and 11.9% higher than FMG alone. To the best of our knowledge, this paper is the first to combine wrist-based PPG, FMG, and ACC signals for hand gesture recognition. These results could serve to inform wrist-based gesture recognition design (e.g., via a smartwatch) and thus expand the capabilities of intuitive and ubiquitous human-machine interaction.

OBJECTIVE: Enable accurate estimation of vertical average loading rate (VALR) in runners with one or more wearable inertial measurement units (IMUs). METHODS: A subject-independent convolutional neural network (CNN) model was developed to estimate VALR from wearable IMUs. Fifteen runners wore IMUs at the trunk, pelvis, thigh, shank, and foot and ran on an instrumented treadmill for combinations of the following conditions: foot-strike (forefoot, mid-foot, rear-foot), step rate (90% to 110% of baseline), running speed (2.4 m/s and 2.8 m/s) and footwear (standard and minimalist running shoes). Thirty-one IMU placement configurations with combinations of one to five IMUs were evaluated. VALR estimations from the wearable IMUs were compared with force-plate VALR measurements. RESULTS: VALR estimations via the subject-independent CNN model with a single shank-worn IMU were highly correlated (ρ = 0.94) with force-plate VALR measurements and were substantially higher than previously reported peak tibial acceleration correlations with force-plate VALR measurements from shank-worn accelerometers (ρ = 0.44-0.66). Correlation results from the CNN model for a single IMU placed at the foot, pelvis, trunk, and thigh were ρ = 0.91, 0.76, 0.69, and 0.65, respectively. There was no improvement in accuracy from the shank-worn IMU when adding 1-4 additional IMUs from the trunk, pelvis, thigh, or foot. CONCLUSION: The proposed subject-independent CNN model with a single shank-worn IMU provides more accurate estimation of VALR than previous wearable sensing approaches. SIGNIFICANCE: This could enable runners to more accurately assess impact loading rates and potentially provide insights into running-related injury risk and prevention.

Effective stroke rehabilitation typically involves improving both motor function and mental state. Traditional rehabilitation systems are generally expensive, not portable and difficult to operate. As a first step toward overcoming these existing shortcomings, we developed a cellphone augmented reality (AR) rehabilitation system for stroke rehabilitation. Three serious games for upper limb motor function and cognitive training were developed in which patients move their arms to touch virtual targets generated in the three-dimensional AR environment. Patients received visual and vibration feedback after successfully touching each target. Eight stroke patients with upper limb dysfunction performed pilot validation testing with the cellphone AR system. Motor function performance was evaluated based on performance in the AR games and mental state was evaluated via questionnaires self-reporting various aspects of mental state. Results showed that motor performance improved by the last trial as compared to the first trial (p<;0.05). In addition, the majority of patients reported that they felt `relaxed' and `happy' while playing the AR rehabilitation games. These results demonstrate the potential of cellphone AR rehabilitation to improve upper-limb motor function and mental state for stroke patients and lay a foundation for a future home-based rehabilitation paradigm.

HaptiVec is a new haptic feedback paradigm for handheld controllers which allows users to feel directional haptic pressure vectors on their fingers and hands while interacting with virtual environments. We embed a 3 by 5 tactile pin array (with an average pin spacing of 25 mm) into the handles of two custom VR type controllers. By presenting directional pressure vectors in eight cardinal directions (N, NE, E, SE, S, SW, W, NW) to users without prior training, they were able to distinguish the correct direction with an accuracy of at least 79%. We illustrate two applications where our device enhances virtual experiences over traditional vibrotactile feedback. In the first application, through the classic first-person shooter Doom, we demonstrate that users can receive directional pressure feedback corresponding to the direction of incident enemy projectiles. In the second application, we demonstrate how our controller can create a more immersive experience by allowing the user to feel their virtual climate by randomizing the directional vectors and presenting the user with "haptic rain" which adapts with the intensity of the rainfall.

Hand gestures are expressive motions that convey meaningful information. The ability for machines to extract and process the underlying meanings of these gestures is critical to many human-interactive applications. Various methods have been proposed, but the development of a more accurate, and simpler system could enable the machine and its user to exchange useful information more effectively. In this paper, a barometric-pressure-sensor-based wristband is presented as an initial proof of such concept. The wristband is composed of an array of 10 barometric pressure sensors spaced evenly around the wrist to estimate pressure profiles as tendons and muscles change with various hand gestures. Subject testing was performed to quantify classification accuracy for three groups of hand gestures: group 1) six wrist gestures, group 2) five single finger flexions, and group 3) ten Chinese number gestures. Leave-one-out cross-validation was used to compute classification accuracy. Results demonstrated classification accuracies of 98% for the wrist gestures, 95% for the single finger flexions, and 90% for Chinese number gestures. The presented pressure sensing wristband could potentially be used for a variety of applications including gesture-controlled devices, health-monitoring devices, and assistive devices for deaf-mute individuals.

While most wearable gesture recognition approaches focus on the forearm or fingers, the wrist may be a more suitable location for practical use. We present the design and validation of a real-time gesture recognition wristband based on surface electromyography and inertial measurement unit sensing fusion, which can recognize 8 air gestures and 4 surface gestures with 2 distinct force levels. Ten healthy subjects performed an initial gesture recognition experiment, followed by a second experiment 1 h later and a third experiment 1 day later. Classification accuracies for the initial experiment were 92.6% and 88.8% for air and surface gestures, respectively, and there were no changes in accuracy results during testing 1 h. and 1 day later (p > 0.05). These results demonstrate the feasibility of wrist-based gesture recognition paving the way for potential future integration in to a smart watch or other wrist-worn wearable for intuitive human computer interaction.

Arm motion timing is critical during basketball shooting. This study used a body-worn, sensorized basketball sleeve to identify arm motion timing characteristics during basketball free throw and jump shot shooting for trained and novice shooters. Current basketball shooting research has typically focused on arm kinematic angles, while shot timing has received comparatively less attention. An experiment was conducted to compare arm motion timing between trained and novice shooters while shooting free throws, and a second experiment compared arm motion timing between free throws and jump shots by trained shooters. Trained shooters shot free throws significantly faster than novice shooters, and trained shooters shot jump shots significantly faster than free throws at the same distance from the basket. Knowledge of arm motion timing characteristics from this study could enable future training for improved shooter accuracy.

Gait retraining is an important rehabilitation method for re-establishing health gait patterns resulting from disease or injury. Optical marker-based motion capture systems are effective for sensing but aren't used widely, due to cost and lack of portability. Moreover, to perform gait retraining, feedback is needed in addition to sensing. This paper presents a wearable sensing and haptic feedback research platform for gait retraining. The platform contains eight distributed nodes (Dots) and a central control unit (Hub) that wirelessly connects to the Dots. Each Dot provides 9-axis inertial sensing and can be configured for sensing or/and providing vibrotactile feedback according to movement training requirements. The Hub receives the sensor data, performs algorithm computation and distributes feedback commands based on the feedback strategy. A foot progression angle (FPA) gait retraining task was performed by six healthy older adults. Participants used the wearable system to learn toe-in gait (foot pointing more inward) and toe-out gait (foot pointing more outward) by adjusting their FPA based on haptic cues to fall within the no feedback zone, i.e. the desired range of acceptable FPAs. After gait retraining, FPA during toe-in gait (1.8±5.6 deg) was significantly higher than during baseline walking (-4.3±5.1 deg) (p<;0.01) and during toe-out gait (-9.9±3.2 deg) (p<;0.01). The no feedback zone was easily found by participants as the percentage of time with no feedback for toe-in gait was 68.3%, and for toe-out gait it was 89.4%. This work demonstrates that the wearable system can be an effective gait retraining research platform.

Psychophysical tests and standardized questionnaires are often used to analyze tactile sensation based on subjective judgment in conventional studies. In contrast with the subjective evaluation, a novel method based on electroencephalography (EEG) is proposed to explore the possibility of quantifying tactile sensation in an objective way. The proposed experiments adopt cutaneous electrical stimulation to generate two kinds of sensations (vibration and pressure) with three grades (low/medium/strong) on eight subjects. Event-related potentials (ERPs) and event-related synchronization/desynchronization (ERS/ERD) are extracted from EEG, which are used as evaluation indexes to distinguish between vibration and pressure, and also to discriminate sensation grades. Results show that five-phase P1–N1–P2–N2–P3 deflection is induced in EEG. Using amplitudes of latter ERP components (N2 and P3), vibration and pressure sensations can be discriminated on both individual and grand-averaged ERP (p < 0.05). The grand-average ERPs can distinguish the three sensations grades, but there is no significant difference on individuals. In addition, ERS/ERD features of mu rhythm (8–13 Hz) are adopted. Vibration and pressure sensations can be discriminated on grand-average ERS/ERD (p < 0.05), but only some individuals show significant difference. The grand-averaged results show that most sensation grades can be differentiated, and most pairwise comparisons show significant difference on individuals (p < 0.05). The work suggests that ERP- and ERS/ERD-based EEG features may have potential to quantify tactile sensations for medical diagnosis or engineering applications.

Gait retraining, a promising treatment for knee osteoarthritis, requires the modification of three separate joint motions. In this paper we present the results of three studies to inform the design of a wearable haptic feedback system for this application. The first study motivates our choice of feedback modality for each of the motions. The latter two studies explore how to present haptic feedback to train three different motions concurrently. When feedback is presented simultaneously, subjects have poor perception of three or more haptic cues, tend to focus on only one motion at a time, and require several steps to modify all three motions. These findings suggest that vibrational feedback should be presented one joint at a time for haptic gait retraining.

This paper introduces a multi-actuator tactile device designed for remote touch communication. While closely-spaced high-frequency vibrotactile actuators can be difficult to distinguish, our system utilized four linear DC motors for presenting spatial tactile messages through low-frequency actuation. An experiment was conducted to determine accuracy for recognizing stimuli presented on the palm of the hand. Participants were asked to identify 10 predefined stimulus patterns created from the four linear actuators positioned in either a diamond or square configuration. Results showed that positional, linear, and circular stimuli were recognized with mean response accuracies of 98.8, 96.5, and 90.2%, respectively. No statistically significant differences were found between the actuator configurations. These findings can be utilized in developing a remote communication channel that supports the transfer of spatial aspects of touch such as mapping the location of finger touch of one user to tactile sensation on the palm of another user.

We present a wearable haptic feedback device for the foot, which gives the sensation of a small pebble in a shoe when actuated and no sensation otherwise. Because it stimulates slowly-adapting as well as fast-adapting mechanoreceptors it is useful for displaying a condition that may persist over time, as well as the occurrence of an event. The feedback, which we call the “virtual pebble” due to its ability to appear on command, is intended as a complement to vibration feedback. We performed a user study to quantify perception accuracy during standing, walking, and jogging for haptic feedback combinations on the foot and knee from vibrotactors and the virtual pebble. We also quantified absolute perception thresholds for single vibration and virtual pebble actuations. Results show that subjects are able to correctly perceive a combination of the pebble and vibration much more accurately than a combination of two vibrations. In addition, subjects are most sensitive to vibration feedback while stationary but most sensitive to virtual pebble feedback while jogging. These findings suggest that the virtual pebble is useful as an additional channel of haptic feedback during ambulatory locomotion.