Abstract Foot‐ground contact information plays a crucial role in character animation and gait analysis, as it helps accurately simulating realistic movement patterns and understanding the biomechanics of walking. Existing motion datasets do not explicitly include foot‐ground contact information, requiring separate computation or manual annotation. Obtaining accurate foot‐ground contact information typically requires additional sensors such as pressure mats or force plates. Without such devices, estimating contact becomes a highly challenging task. We propose ContactVision, a deep learning framework that detects heel and toe contact states directly from video. Our network is trained in a supervised manner using contact labels derived from motion capture data via ground reaction force estimation. This enables training on existing datasets without the need for additional hardware. We demonstrate the utility of our contact detection network in two downstream tasks: gait motion reconstruction and gait analysis. For animation, we incorporate predicted contact labels into a reinforcement learning framework with a two‐segment foot model, enabling realistic foot articulation behavior. For analysis, we estimate clinically relevant gait parameters such as double and single support times, and validate the accuracy against pressure sensor mat data and prior video‐based methods. Our results show competitive performance in both animation and analysis settings. The code is publicly available at github.com/DaeeYong/ContactVision .

We propose a new method that reconstructs 3D human motion from in-the-wild video by making full use of prior knowledge on the laws of physics. Previous studies focus on reconstructing joint angles and positions in the body local coordinate frame. Body translations and rotations in the global reference frame are partially reconstructed only when the video has a static camera view. We are interested in overcoming this static view limitation to deal with dynamic view videos. The camera may pan, tilt, and zoom to track the moving subject. Since we do not assume any limitations on camera movements, body translations and rotations from the video do not correspond to absolute positions in the reference frame. The key technical challenge is inferring body translations and rotations from a sequence of 3D full-body poses, assuming the absence of root motion. This inference is possible because human motion obeys the law of physics. Our reconstruction algorithm produces a control policy that simulates 3D human motion imitating the one in the video. Our algorithm is particularly useful for reconstructing highly dynamic movements, such as sports, dance, gymnastics, and parkour actions.

Abstract Realistic modelling of a human‐like character is one of the main topics in computer graphics to simulate human motion physically and also look realistically. Of the body parts, a human foot interacts with the ground, and plays an essential role in weight transmission, balancing posture and assisting ambulation. However, in the previous researches, the foot model was often simplified into one or two rigid bodies connected by a revolute joint. We propose a new foot model consisting of multiple segments to reproduce human foot shape and its functionality accurately. Based on the new model, we develop a foot pose controller that can reproduce foot postures that are generally not obtained in motion capture data. We demonstrate the validity of our foot model and the effectiveness of our foot controller with a variety of foot motions in a physics‐based simulation.

Abstract Figure skating is one of the most popular ice sports at the Winter Olympic Games. The skaters perform several skating skills to express the beauty of the art on ice. Skating involves moving on ice while wearing skate shoes with thin blades; thus, it requires much practice to skate without losing balance. Moreover, figure skating presents dynamic moves, such as jumping, artistically. Therefore, demonstrating figure skating skills is even more difficult to achieve than basic skating, and professional skaters often fall during Winter Olympic performances. We propose a system to demonstrate figure skating motions with a physically simulated human‐like character. We simulate skating motions with non‐holonomic constraints, which make the skate blade glide on the ice surface. It is difficult to obtain reference motions from figure skaters because figure skating motions are very fast and dynamic. Instead of using motion capture data, we use key poses extracted from videos on YouTube and complete reference motions using trajectory optimization. We demonstrate figure skating skills, such as crossover, three‐turn, and even jump. Finally, we use deep reinforcement learning to generate a robust controller for figure skating skills.

We present a multi-segment foot model and its control method for the simulation of realistic bipedal behaviors. The ground reaction force is the only source of control for a biped that stands and walks on its feet. The foot is the body part that interacts with the ground and produces appropriate actuation to the body. Foot anatomy features 26 bones and many more muscles that play an important role in weight transmission, balancing posture and assisting ambulation. Previously, the foot model was often simplified into one or two rigid bodies connected by a revolute joint. We propose a new foot model consisting of multiple segments to accurately reproduce human foot shape and its functionality. Based on the new model, we developed a foot pose controller that can reproduce foot postures that are generally not obtained in motion capture data. We demonstrate the validity of our foot model and the effectiveness of our foot controller with a variety of foot motions in a physics-based simulation.

We propose a framework for simulation and control of the human musculoskeletal system, capable of reproducing realistic animations of dexterous activities with high-level coordination. We present the first controllable system in this class that incorporates volumetric muscle actuators, tightly coupled with the motion controller, in enhancement of line-segment approximations that prior art is overwhelmingly restricted to. The theoretical framework put forth by our methodology computes all the necessary Jacobians for control, even with the drastically increased dimensionality of the state descriptors associated with three-dimensional, volumetric muscles. The direct coupling of volumetric actuators in the controller allows us to model muscular deficiencies that manifest in shape and geometry, in ways that cannot be captured with line-segment approximations. Our controller is coupled with a trajectory optimization framework, and its efficacy is demonstrated in complex motion tasks such as juggling, and weightlifting sequences with variable anatomic parameters and interaction constraints.