To improve vision-based methods for understanding how people interact with their physical environment, we introduce a multi-view video and body-contact sensing dataset designed to capture dynamic human activities that involve interactions with the physical environment. The dataset includes activities such as parkour, physical training, and gym exercises, characterized by frequent body-environment contact. The proposed dataset includes 780 K images across 120 K pose sequences from 7 subjects. Each subject is captured by 6 synchronized third-person cameras, a single egocentric camera, and multiple contact sensors worn on the body. Using our proposed dataset, we benchmark state-of-the-art vision-based body contact models and show significant limitations in existing methods. Furthermore, we benchmark existing human pose estimation methods on our dataset and show that they fail under significant occlusion caused by close interactions with the environment, which indicates that our dataset can also be used to further develop pose estimation models to be more robust during interaction with the environment. To facilitate better human pose estimation from video, we introduce and evaluate a video-based human contact detection model that outperforms existing imagebased methods, underscoring the potential improvements from integrating contact information into pose estimation models. See the project page at: https://yohanshin.github.io/bodycontact4d.github.io/

Accurate three-dimensional (3D) human motion tracking is essential for immersive augmented reality$(A R)$and virtual reality (VR) applications, allowing users to engage with virtual environments through realistic full-body avatars. Achieving this level of detail, however, is challenging when the driving signals are sparse, typically coming only from upper-body sensors, such as head-mounted devices and hand controllers. To address this challenge, we propose EgoMDM (Egocentric Motion Diffusion Model), an end-to-end diffusion-based framework designed to reconstruct full-body motion from sparse tracking signals. EgoMDM models human motion in a conditional autoregressive manner using a unidirectional recurrent neural network, making it well-suited for real-time applications. By embedding local-to-global translation, forward and inverse kinematics, and foot-contact detection within the diffusion framework, EgoMDM achieves seamless, end-to-end motion synthesis, effectively reducing artifacts like foot sliding and ground penetration. Additionally, EgoMDM is conditioned on the user's body scale, allowing it to generalize across a diverse population and produce consistent avatar shapes over time. In our extensive experiments on the AMASS motion capture dataset, EgoMDM achieves state-of-the-art performance in both motion tracking accuracy and synthesis quality, demonstrating its robustness and adaptability across various human motion scenarios. Furthermore, EgoMDM significantly outperforms the existing models when tested on real signal inputs, highlighting its robustness and applicability to the real-world data. See the project page at: https://yohanshin.github.io/egomdm.github.io/

In this paper, our goal is to enable quantitative feedback on muscle fatigue during exercise to optimize exercise effectiveness while minimizing injury risk. We seek to capture fatigue by monitoring surface vibrations that muscle exertion induces. Muscle vibrations are unique as they arise from the asynchronous firing of motor units, producing surface micro-displacements that are broadband, nonlinear, and seemingly stochastic. Accurately sensing these noise-like signals requires new algorithmic strategies that can uncover their underlying structure. We present GigaFlex the first contactless system that measures muscle vibrations using mmWave radar to infer muscle force and detect fatigue. GigaFlex draws on algorithmic foundations from Chaos theory to model the deterministic patterns of muscle vibrations and extend them to the radar domain. Specifically, we design a radar processing architecture that systematically infuses principles from Chaos theory and nonlinear dynamics throughout the sensing pipeline, spanning localization, segmentation, and learning, to estimate muscle forces during static and dynamic weight-bearing exercises. Across a 23-participant study, GigaFlex estimates maximum voluntary isometric contraction (MVIC) root mean square error (RMSE) of 5.9\%, and detects one to three Repetitions in Reserve (RIR), a key quantitative muscle fatigue metric, with an AUC of 0.83 to 0.86, performing comparably to a contact-based IMU baseline. Our system can enable timely feedback that can help prevent fatigue-induced injury, and opens new opportunities for physiological sensing of complex, non-periodic biosignals.

The estimation of 3D human motion from video has progressed rapidly but current methods still have several key limitations. First, most methods estimate the human in camera coordinates. Second, prior work on estimating humans in global coordinates often assumes a flat ground plane and produces foot sliding. Third, the most accurate methods rely on computationally expensive optimization pipelines, limiting their use to offline applications. Finally, existing video-based methods are surprisingly less accurate than singleframe methods. We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video. WHAM learns to lift 2D keypoint sequences to 3D using motion capture data and fuses this with video features, integrating motion context and visual information. WHAM exploits camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. We combine this with a contact-aware trajectory refinement method that lets WHAM capture human motion in diverse conditions, such as climbing stairs. WHAM outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks. Code is available for research purposes at http://wham.is.tue.mpg.del.

OBJECTIVE: Exercise monitoring with low-cost wearables could improve the efficacy of remote physical-therapy prescriptions by tracking compliance and informing the delivery of tailored feedback. While a multitude of commercial wearables can detect activities of daily life, such as walking and running, they cannot accurately detect physical-therapy exercises. The goal of this study was to build open-source classifiers for remote physical-therapy monitoring and provide insight on how data collection choices may impact classifier performance. METHODS: We trained and evaluated multi-class classifiers using data from 19 healthy adults who performed 37 exercises while wearing 10 inertial measurement units (IMUs) on the chest, pelvis, wrists, thighs, shanks, and feet. We investigated the effect of sensor density, location, type, sampling frequency, output granularity, feature engineering, and training-data size on exercise-classification performance. RESULTS: Exercise groups (n = 10) could be classified with 96% accuracy using a set of 10 IMUs and with 89% accuracy using a single pelvis-worn IMU. Multiple sensor modalities (i.e., accelerometers and gyroscopes), high sampling frequencies, and more data from the same population did not improve model performance, but in the future data from diverse populations and better feature engineering could. CONCLUSIONS: Given the growing demand for exercise monitoring systems, our sensitivity analyses, along with open-source tools and data, should reduce barriers for product developers, who are balancing accuracy with product formfactor, and increase transparency and trust in clinicians and patients.

Wearable sensing and computer vision could move biomechanics from specialized laboratories to natural environments, but better algorithms are needed to extract meaningful outcomes from these emerging modalities. In this article, we present new models for estimating biomechanical outcomes—the knee adduction moment (KAM) and knee flexion moment (KFM)—from fusion of smartphone cameras and wearable inertial measurement units (IMUs) among young healthy nonobese males. A deep learning model was developed to extract features, fuse multimodal data, and estimate KAM and KFM. Walking data from 17 subjects were recorded with eight IMUs and two smartphone cameras. The model that used IMU-camera fusion was significantly more accurate than those using IMUs or cameras alone. The root-mean-square errors of the fusion model were 0.49$\%\;\mathbf {BW}\cdot \mathbf {BH}$for KAM and 0.66$\%\;\mathbf {BW}\cdot \mathbf {BH}$for KFM estimation, which are lower than clinically significant thresholds. With larger and more diverse data, this model could enable assessment of knee moments in clinics and homes.

Knee osteoarthritis is a progressive disease mediated by high joint loads. Foot progression angle modifications that reduce the knee adduction moment (KAM), a surrogate of knee loading, have demonstrated efficacy in alleviating pain and improving function. Although changes to the foot progression angle are overall beneficial, KAM reductions are not consistent across patients. Moreover, customized interventions are time-consuming and require instrumentation not commonly available in the clinic. We present a regression model that uses minimal clinical data-a set of six features easily obtained in the clinic-to predict the extent of first peak KAM reduction after toe-in gait retraining. For such a model to generalize, the training data must be large and variable. Given the lack of large public datasets that contain different gaits for the same patient, we generated this dataset synthetically. Insights learned from a ground-truth dataset with both baseline and toe-in gait trials (N = 12) enabled the creation of a large (N = 138) synthetic dataset for training the predictive model. On a test set of data collected by a separate research group (N = 15), the first peak KAM reduction was predicted with a mean absolute error of 0.134% body weight * height (%BW*HT). This error is smaller than the standard deviation of the first peak KAM during baseline walking averaged across test subjects (0.306%BW*HT). This work demonstrates the feasibility of training predictive models with synthetic data and provides clinicians with a new tool to predict the outcome of patient-specific gait retraining without requiring gait lab instrumentation.