Reconstructing 3D human movements from video sequences is an important task in the fields of computer vision, graphics, and biomechanics. Although much progress has been made to infer 3D human mesh based on visual contexts provided in video sequences, generalization to in-the-wild videos still remains challenging for existing human mesh recovery (HMR) methods. To overcome inaccurate prediction, they can perform a second step optimization that refines the inaccurate estimations continuously at test time. Most optimization methods seek fitting of the body joints in the image space with respect to pseudo ground truth predicted by an off-the-shelf key point detector. However, state-of-theart detectors still introduce errors, especially for challenging poses. In this work, we rethink the dependency on the 2D key point fitting paradigm and present NeuHMR, an optimization-based mesh recovery framework based on recent advances in neural rendering. Our method builds on Human Neural Radiance Fields that allow the refinement of human meshes through animatable$2 D$renderings. We evaluated our method on two common benchmarks and validated its effectiveness.

Recent advances in text-to-image diffusion models have been driven by the increasing availability of paired 2D data. However, the development of 3D diffusion models has been hindered by the scarcity of high-quality 3D data, resulting in less competitive performance compared to their 2D counterparts. To address this challenge, we propose repurposing pre-trained 2D diffusion models for 3D object generation. We introduce Gaussian Atlas, a novel representation that utilizes dense 2D grids, enabling the fine-tuning of 2D diffusion models to generate 3D Gaussians. Our approach demonstrates successful transfer learning from a pre-trained 2D diffusion model to a 2D manifold flattened from 3D structures. To support model training, we compile GaussianVerse, a large-scale dataset comprising 205K high-quality 3D Gaussian fittings of various 3D objects. Our experimental results show that text-to-image diffusion models can be effectively adapted for 3D content generation, bridging the gap between 2D and 3D modeling.

Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However, despite the inherent connection between pose estimation and biomechanics, these disciplines have largely remained disparate. For example, most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error, Percentage of Correct Keypoints, or mean Average Precision to assess performance, without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge, we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation, evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench, we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge, we introduce SynthPose, a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such fine-tuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover, OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall, OpenCapBench bridges the computer vision and biomechanics communities, aiming to drive simultaneous advances in both areas.

For individuals with knee osteoarthritis, increased knee loading is linked to disease progression and pain. Some approaches to treating osteoarthritis, such as specialized footwear, braces, and powered exoskeletons, also increase the mass of the lower limbs, which could lead to increases in knee loads. Prior studies have investigated the effect of changes in torso mass and total body mass on peak knee contact forces, but the effects of increased leg mass remain unclear. In this study, we created musculoskeletal simulations informed by experimental data to estimate tibiofemoral knee contact force under different lower-limb segment mass conditions. The mass of the foot, shank, and thigh were varied by adding weights to each segment, separately and concurrently, as healthy young adults (N = 10) walked on a treadmill. Kinematics, kinetics, and muscle activity were recorded. Our simulations used an optimal control framework that enforced experimental kinematics while minimizing a combination of net joint moment errors and mismatch between measured and estimated muscle activity. The simulations revealed that adding mass to the lower-limb segments linearly increased early- and late-stance peaks in knee contact force, but that the slope of this relationship was different for each peak and each mass placement location. For each 1% of body weight (BW) added per limb (2% BW total) at the thigh, shank, and foot, early-stance peak knee contact force increased by 1.5%, 2.1%, and 5.9% (r = 0.71), while late-stance peak contact force increased by 1.6%, 0.9% and 3.0% (r = 0.67), respectively. Adding mass to the thigh and shank increases peak contact force at or below the rate of increase in body mass, while adding mass to the foot disproportionately increases peak knee contact force. These detrimental effects should be considered when designing interventions for osteoarthritis.

Analyzing anatomic shapes of tissues and organs is pivotal for accurate disease diagnostics and clinical decision-making. One prominent disease that depends on anatomic shape analysis is osteoarthritis, which affects 30 million Americans. To advance osteoarthritis diagnostics and prognostics, we introduce ShapeMed-Knee, a 3D shape dataset with 9,376 high-resolution, medical-imaging-based 3D shapes of both femur bone and cartilage. Besides data, ShapeMed-Knee includes two benchmarks for assessing reconstruction accuracy and five clinical prediction tasks that assess the utility of learned shape representations. Leveraging ShapeMed-Knee, we develop and evaluate a novel hybrid explicit-implicit neural shape model which achieves up to 40% better reconstruction accuracy than a statistical shape model and two implicit neural shape models. Our hybrid models achieve state-of-the-art performance for preserving cartilage biomarkers (root mean squared error ≤ 0.05 vs. ≤ 0.07, 0.10, and 0.14). Our models are also the first to successfully predict localized structural features of osteoarthritis, outperforming shape models and convolutional neural networks applied to raw magnetic resonance images and segmentations (e.g., osteophyte size and localization 63% accuracy vs. 49-61%). The ShapeMed-Knee dataset provides medical evaluations to reconstruct multiple anatomic surfaces and embed meaningful disease-specific information. ShapeMed-Knee reduces barriers to applying 3D modeling in medicine, and our benchmarks highlight that advancements in 3D modeling can enhance the diagnosis and risk stratification for complex diseases. The dataset, code, and benchmarks are freely accessible.

Motion capture from a limited number of body-worn sensors, such as inertial measurement units (IMUs) and pressure insoles, has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. While a common goal across applications is to use the minimal number of sensors to achieve required accuracy, the optimal arrangement of the sensors might differ from application to application. We propose a single diffusion model, Dif-fusionPoser, which reconstructs human motion in real-time from an arbitrary combination of sensors, including IMUs placed at specified locations, and, pressure insoles. Un-like existing methods, our model grants users the flexibility to determine the number and arrangement of sensors tai-lored to the specific activity of interest, without the need for retraining. A novel autoregressive inferencing scheme ensures real-time motion reconstruction that closely aligns with measured sensor signals. The generative nature of Dif-fusionl'oser ensures realistic behavior, even for degrees-of-freedom not directly measured. Qualitative results can be found on our website: https://diffusionposer.github.io/.

Existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervisions by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and found that it achieves state-of-the-art performance in the rendering of occluded humans.

Musculoskeletal geometry and muscle volumes vary widely in the population and are intricately linked to the performance of tasks ranging from walking and running to jumping and sprinting. As an alternative to experimental approaches, where it is difficult to isolate factors and establish causal relationships, simulations can be used to independently vary musculoskeletal geometry and muscle volumes, and develop a fundamental understanding. However, our ability to understand how these parameters affect task performance has been limited due to the high computational cost of modelling the necessary complexity of the musculoskeletal system and solving the requisite multi-dimensional optimization problem. For example, sprinting and running are fundamental to many forms of sport, but past research on the relationships between musculoskeletal geometry, muscle volumes, and running performance has been limited to observational studies, which have not established cause-effect relationships, and simulation studies with simplified representations of musculoskeletal geometry. In this study, we developed a novel musculoskeletal simulator that is differentiable with respect to musculoskeletal geometry and muscle volumes. This simulator enabled us to find the optimal body segment dimensions and optimal distribution of added muscle volume for sprinting and marathon running. Our simulation results replicate experimental observations, such as increased muscle mass in sprinters, as well as a mass in the lower end of the healthy BMI range and a higher leg-length-to-height ratio in marathon runners. The simulations also reveal new relationships, for example showing that hip musculature is vital to both sprinting and marathon running. We found hip flexor and extensor moment arms were maximized to optimize sprint and marathon running performance, and hip muscles the main target when we simulated strength training for sprinters. Our simulation results provide insight to inspire future studies to examine optimal strength training. Our simulator can be extended to other athletic tasks, such as jumping, or to non-athletic applications, such as designing interventions to improve mobility in older adults or individuals with movement disorders.

Walking balance is central to independent mobility, and falls due to loss of balance are a leading cause of death for people 65 years of age and older. Bipedal gait is typically unstable, but healthy humans use corrective torques to counteract perturbations and stabilize gait. Exoskeleton assistance could benefit people with neuromuscular deficits by providing stabilizing torques at lower-limb joints to replace lost muscle strength and sensorimotor control. However, it is unclear how applied exoskeleton torques translate to changes in walking kinematics. This study used musculoskeletal simulation to investigate how exoskeleton torques applied to the ankle and subtalar joints alter center of mass kinematics during walking. We first created muscle-driven walking simulations using OpenSim Moco by tracking experimental kinematics and ground reaction forces recorded from five healthy adults. We then used forward integration to simulate the effect of exoskeleton torques applied to the ankle and subtalar joints while keeping muscle excitations fixed based on our previous tracking simulation results. Exoskeleton torque lasted for 15% of the gait cycle and was applied between foot-flat and toe-off during the stance phase, and changes in center of mass kinematics were recorded when the torque application ended. We found that changes in center of mass kinematics were dependent on both the type and timing of exoskeleton torques. Plantarflexion torques produced upward and backward changes in velocity of the center of mass in mid-stance and upward and smaller forward velocity changes near toe-off. Eversion and inversion torques primarily produced lateral and medial changes in velocity in mid-stance, respectively. Intrinsic muscle properties reduced kinematic changes from exoskeleton torques. Our results provide mappings between ankle plantarflexion and inversion-eversion torques and changes in center of mass kinematics which can inform designers building exoskeletons aimed at stabilizing balance during walking. Our simulations and software are freely available and allow researchers to explore the effects of applied torques on balance and gait.

Measures of human movement dynamics can predict outcomes like injury risk or musculoskeletal disease progression. However, these measures are rarely quantified in large-scale research studies or clinical practice due to the prohibitive cost, time, and expertise required. Here we present and validate OpenCap, an open-source platform for computing both the kinematics (i.e., motion) and dynamics (i.e., forces) of human movement using videos captured from two or more smartphones. OpenCap leverages pose estimation algorithms to identify body landmarks from videos; deep learning and biomechanical models to estimate three-dimensional kinematics; and physics-based simulations to estimate muscle activations and musculoskeletal dynamics. OpenCap's web application enables users to collect synchronous videos and visualize movement data that is automatically processed in the cloud, thereby eliminating the need for specialized hardware, software, and expertise. We show that OpenCap accurately predicts dynamic measures, like muscle activations, joint loads, and joint moments, which can be used to screen for disease risk, evaluate intervention efficacy, assess between-group movement differences, and inform rehabilitation decisions. Additionally, we demonstrate OpenCap's practical utility through a 100-subject field study, where a clinician using OpenCap estimated musculoskeletal dynamics 25 times faster than a laboratory-based approach at less than 1% of the cost. By democratizing access to human movement analysis, OpenCap can accelerate the incorporation of biomechanical metrics into large-scale research studies, clinical trials, and clinical practice.

Great progress has been made in estimating 3D human pose and shape from images and video by training neural networks to directly regress the parameters of parametric human models like SMPL. However, existing body models have simplified kinematic structures that do not correspond to the true joint locations and articulations in the human skeletal system, limiting their potential use in biomechanics. On the other hand, methods for estimating biomechanically accurate skeletal motion typically rely on complex motion capture systems and expensive optimization methods. What is needed is a parametric 3D human model with a biomechanically accurate skeletal structure that can be easily posed. To that end, we develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton. To enable this, we need training data of skeletons inside SMPL meshes in diverse poses. We build such a dataset by optimizing biomechanically accurate skeletons inside SMPL meshes from AMASS sequences. We then learn a regressor from SMPL mesh vertices to the optimized joint locations and bone rotations. Finally, we re-parametrize the SMPL mesh with the new kinematic parameters. The resulting SKEL model is animatable like SMPL but with fewer, and biomechanically-realistic, degrees of freedom. We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods. By fitting SKEL to SMPL meshes we are able to "upgrade" existing human pose and shape datasets to include biomechanical parameters. SKEL provides a new tool to enable biomechanics in the wild, while also providing vision and graphics researchers with a better constrained and more realistic model of human articulation. The model, code, and data are available for research at https://skel.is.tue.mpg.de.

Musculoskeletal simulations are used in many different applications, ranging from the design of wearable robots that interact with humans to the analysis of patients with impaired movement. Here, we introduce OpenSim Moco, a software toolkit for optimizing the motion and control of musculoskeletal models built in the OpenSim modeling and simulation package. OpenSim Moco uses the direct collocation method, which is often faster and can handle more diverse problems than other methods for musculoskeletal simulation. Moco frees researchers from implementing direct collocation themselves-which typically requires extensive technical expertise-and allows them to focus on their scientific questions. The software can handle a wide range of problems that interest biomechanists, including motion tracking, motion prediction, parameter optimization, model fitting, electromyography-driven simulation, and device design. Moco is the first musculoskeletal direct collocation tool to handle kinematic constraints, which enable modeling of kinematic loops (e.g., cycling models) and complex anatomy (e.g., patellar motion). To show the abilities of Moco, we first solved for muscle activity that produced an observed walking motion while minimizing squared muscle excitations and knee joint loading. Next, we predicted how muscle weakness may cause deviations from a normal walking motion. Lastly, we predicted a squat-to-stand motion and optimized the stiffness of an assistive device placed at the knee. We designed Moco to be easy to use, customizable, and extensible, thereby accelerating the use of simulations to understand the movement of humans and other animals.

Deficits in the ankle plantarflexor muscles, such as weakness and contracture, occur commonly in conditions such as cerebral palsy, stroke, muscular dystrophy, Charcot-Marie-Tooth disease, and sarcopenia. While these deficits likely contribute to observed gait pathologies, determining cause-effect relationships is difficult due to the often co-occurring biomechanical and neural deficits. To elucidate the effects of weakness and contracture, we systematically introduced isolated deficits into a musculoskeletal model and generated simulations of walking to predict gait adaptations due to these deficits. We trained a planar model containing 9 degrees of freedom and 18 musculotendon actuators to walk using a custom optimization framework through which we imposed simple objectives, such as minimizing cost of transport while avoiding falling and injury, and maintaining head stability. We first generated gaits at prescribed speeds between 0.50 m/s and 2.00 m/s that reproduced experimentally observed kinematic, kinetic, and metabolic trends for walking. We then generated a gait at self-selected walking speed; quantitative comparisons between our simulation and experimental data for joint angles, joint moments, and ground reaction forces showed root-mean-squared errors of less than 1.6 standard deviations and normalized cross-correlations above 0.8 except for knee joint moment trajectories. Finally, we applied mild, moderate, and severe levels of muscle weakness or contracture to either the soleus (SOL) or gastrocnemius (GAS) or both of these major plantarflexors (PF) and retrained the model to walk at a self-selected speed. The model was robust to all deficits, finding a stable gait in all cases. Severe PF weakness caused the model to adopt a slower, "heel-walking" gait. Severe contracture of only SOL or both PF yielded similar results: the model adopted a "toe-walking" gait with excessive hip and knee flexion during stance. These results highlight how plantarflexor weakness and contracture may contribute to observed gait patterns.

Abstract Motion capture is often retargeted to new, and sometimes drastically different, characters. When the characters take on realistic human shapes, however, we become more sensitive to the motion looking right. This means adapting it to be consistent with the physical constraints imposed by different body shapes. We show how to take realistic 3D human shapes, approximate them using a simplified representation, and animate them so that they move realistically using physically‐based retargeting. We develop a novel spacetime optimization approach that learns and robustly adapts physical controllers to new bodies and constraints. The approach automatically adapts the motion of the mocap subject to the body shape of a target subject. This motion respects the physical properties of the new body and every body shape results in a different and appropriate movement. This makes it easy to create a varied set of motions from a single mocap sequence by simply varying the characters. In an interactive environment, successful retargeting requires adapting the motion to unexpected external forces. We achieve robustness to such forces using a novel LQR‐tree formulation. We show that the simulated motions look appropriate to each character's anatomy and their actions are robust to perturbations.

Movement is fundamental to human and animal life, emerging through interaction of complex neural, muscular, and skeletal systems. Study of movement draws from and contributes to diverse fields, including biology, neuroscience, mechanics, and robotics. OpenSim unites methods from these fields to create fast and accurate simulations of movement, enabling two fundamental tasks. First, the software can calculate variables that are difficult to measure experimentally, such as the forces generated by muscles and the stretch and recoil of tendons during movement. Second, OpenSim can predict novel movements from models of motor control, such as kinematic adaptations of human gait during loaded or inclined walking. Changes in musculoskeletal dynamics following surgery or due to human-device interaction can also be simulated; these simulations have played a vital role in several applications, including the design of implantable mechanical devices to improve human grasping in individuals with paralysis. OpenSim is an extensible and user-friendly software package built on decades of knowledge about computational modeling and simulation of biomechanical systems. OpenSim's design enables computational scientists to create new state-of-the-art software tools and empowers others to use these tools in research and clinical applications. OpenSim supports a large and growing community of biomechanics and rehabilitation researchers, facilitating exchange of models and simulations for reproducing and extending discoveries. Examples, tutorials, documentation, and an active user forum support this community. The OpenSim software is covered by the Apache License 2.0, which permits its use for any purpose including both nonprofit and commercial applications. The source code is freely and anonymously accessible on GitHub, where the community is welcomed to make contributions. Platform-specific installers of OpenSim include a GUI and are available on simtk.org.

Regular physical activity helps prevent heart disease, stroke, diabetes, and other chronic diseases, yet a broad range of conditions impair mobility at great personal and societal cost. Vast amounts of data characterizing human movement are available from research labs, clinics, and millions of smartphones and wearable sensors, but integration and analysis of this large quantity of mobility data are extremely challenging. The authors have established the Mobilize Center (http://mobilize.stanford.edu) to harness these data to improve human mobility and help lay the foundation for using data science methods in biomedicine. The Center is organized around 4 data science research cores: biomechanical modeling, statistical learning, behavioral and social modeling, and integrative modeling. Important biomedical applications, such as osteoarthritis and weight management, will focus the development of new data science methods. By developing these new approaches, sharing data and validated software tools, and training thousands of researchers, the Mobilize Center will transform human movement research.

Physics-based simulation provides a powerful framework for understanding biological form and function. Simulations can be used by biologists to study macromolecular assemblies and by clinicians to design treatments for diseases. Simulations help biomedical researchers understand the physical constraints on biological systems as they engineer novel drugs, synthetic tissues, medical devices, and surgical interventions. Although individual biomedical investigators make outstanding contributions to physics-based simulation, the field has been fragmented. Applications are typically limited to a single physical scale, and individual investigators usually must create their own software. These conditions created a major barrier to advancing simulation capabilities. In 2004, we established a National Center for Physics-Based Simulation of Biological Structures (Simbios) to help integrate the field and accelerate biomedical research. In 6 years, Simbios has become a vibrant national center, with collaborators in 16 states and eight countries. Simbios focuses on problems at both the molecular scale and the organismal level, with a long-term goal of uniting these in accurate multiscale simulations.

We present a technique for automatically synthesizing walking and running controllers for physically-simulated 3D humanoid characters. The sagittal hip, knee, and ankle degrees-of-freedom are actuated using a set of eight Hill-type musculotendon models in each leg, with biologically-motivated control laws. The parameters of these control laws are set by an optimization procedure that satisfies a number of locomotion task terms while minimizing a biological model of metabolic energy expenditure. We show that the use of biologically-based actuators and objectives measurably increases the realism of gaits generated by locomotion controllers that operate without the use of motion capture data, and that metabolic energy expenditure provides a simple and unifying measurement of effort that can be used for both walking and running control optimization.

Physics-based simulation is needed to understand the function of biological structures and can be applied across a wide range of scales, from molecules to organisms. Simbios (the National Center for Physics-Based Simulation of Biological Structures, http://www.simbios.stanford.edu/) is one of seven NIH-supported National Centers for Biomedical Computation. This article provides an overview of the mission and achievements of Simbios, and describes its place within systems biology. Understanding the interactions between various parts of a biological system and integrating this information to understand how biological systems function is the goal of systems biology. Many important biological systems comprise complex structural systems whose components interact through the exchange of physical forces, and whose movement and function is dictated by those forces. In particular, systems that are made of multiple identifiable components that move relative to one another in a constrained manner are multibody systems. Simbios' focus is creating methods for their simulation. Simbios is also investigating the biomechanical forces that govern fluid flow through deformable vessels, a central problem in cardiovascular dynamics. In this application, the system is governed by the interplay of classical forces, but the motion is distributed smoothly through the materials and fluids, requiring the use of continuum methods. In addition to the research aims, Simbios is working to disseminate information, software and other resources relevant to biological systems in motion.