Learning deep neural networks on human motion data has become common in computer graphics research, but the heterogeneity of available datasets poses challenges for training large-scale networks. This paper presents a framework that allows us to solve various animation tasks in a skeleton-agnostic manner. The core of our framework is to learn an embedding space to disentangle skeleton-related information from input motion while preserving semantics, which we call Skeleton-Agnostic Motion Embedding (SAME). To efficiently learn the embedding space, we develop a novel autoencoder with graph convolution networks and provide new formulations of various animation tasks operating in the SAME space. We showcase various examples, including retargeting, reconstruction, and interactive character control, and conduct an ablation study to validate design choices made during development.

The Chimera is a mythological hybrid creature composed of different animal parts. The chimera's movements are highly dependent on the spatial and temporal alignments of its composing parts. In this paper, we present a novel algorithm that creates and animates chimeras by dynamically reassembling source characters and their movements. Our algorithm exploits a two-network architecture: part assembler and dynamic controller. The part assembler is a supervised learning layer that searches for the spatial alignment among body parts, assuming that the temporal alignment is provided. The dynamic controller is a reinforcement learning layer that learns robust control policy for a wide variety of potential temporal alignments. These two layers are tightly intertwined and learned simultaneously. The chimera animation generated by our algorithm is energy efficient and expressive in terms of describing weight shifting, balancing, and full-body coordination. We demonstrate the versatility of our algorithm by generating the motor skills of a large variety of chimeras from limited source characters.

Abstract We present a novel retargeting algorithm that transfers the musculature of a reference anatomical model to new bodies with different sizes, body proportions, muscle capability, and joint range of motion while preserving the functionality of the original musculature as closely as possible. The geometric configuration and physiological parameters of musculotendon units are estimated and optimized to adapt to new bodies. The range of motion around joints is estimated from a motion capture dataset and edited further for individual models. The retargeted model is simulation‐ready, so we can physically simulate muscle‐actuated motor skills with the model. Our system is capable of generating a wide variety of anatomical bodies that can be simulated to walk, run, jump and dance while maintaining balance under gravity. We will also demonstrate the construction of individualized musculoskeletal models from bi‐planar X‐ray images and medical examination.

We present a new algorithm that learns a parameterized family of motor skills from a single motion clip. The motor skills are represented by a deep policy network, which produces a stream of motions in physics simulation in response to user input and environment interaction by navigating continuous action space. Three novel technical components play an important role in the success of our algorithm. First, it explicitly constructs motion parameterization that maps action parameters to their corresponding motions. Simultaneous learning of motion parameterization and motor skills significantly improves the performance and visual quality of learned motor skills. Second, continuous-time reinforcement learning is adopted to explore temporal variations as well as spatial variations in motion parameterization. Lastly, we present a new automatic curriculum generation method that explores continuous action space more efficiently. We demonstrate the flexibility and versatility of our algorithm with highly dynamic motor skills that can be parameterized by task goals, body proportions, physical measurements, and environmental conditions.

Creating agile and responsive characters from a collection of unorganized human motion has been an important problem of constructing interactive virtual environments. Recently, learning-based approaches have successfully been exploited to learn deep network policies for the control of interactive characters. The agility and responsiveness of deep network policies are influenced by many factors, such as the composition of training datasets, the architecture of network models, and learning algorithms that involve many threshold values, weights, and hyper-parameters. In this paper, we present a novel teacher-student framework to learn time-critically responsive policies, which guarantee the time-to-completion between user inputs and their associated responses regardless of the size and composition of the motion databases. We demonstrate the effectiveness of our approach with interactive characters that can respond to the user's control quickly while performing agile, highly dynamic movements.

We propose a new method that reconstructs 3D human motion from in-the-wild video by making full use of prior knowledge on the laws of physics. Previous studies focus on reconstructing joint angles and positions in the body local coordinate frame. Body translations and rotations in the global reference frame are partially reconstructed only when the video has a static camera view. We are interested in overcoming this static view limitation to deal with dynamic view videos. The camera may pan, tilt, and zoom to track the moving subject. Since we do not assume any limitations on camera movements, body translations and rotations from the video do not correspond to absolute positions in the reference frame. The key technical challenge is inferring body translations and rotations from a sequence of 3D full-body poses, assuming the absence of root motion. This inference is possible because human motion obeys the law of physics. Our reconstruction algorithm produces a control policy that simulates 3D human motion imitating the one in the video. Our algorithm is particularly useful for reconstructing highly dynamic movements, such as sports, dance, gymnastics, and parkour actions.

Abstract Realistic modelling of a human‐like character is one of the main topics in computer graphics to simulate human motion physically and also look realistically. Of the body parts, a human foot interacts with the ground, and plays an essential role in weight transmission, balancing posture and assisting ambulation. However, in the previous researches, the foot model was often simplified into one or two rigid bodies connected by a revolute joint. We propose a new foot model consisting of multiple segments to reproduce human foot shape and its functionality accurately. Based on the new model, we develop a foot pose controller that can reproduce foot postures that are generally not obtained in motion capture data. We demonstrate the validity of our foot model and the effectiveness of our foot controller with a variety of foot motions in a physics‐based simulation.

Abstract Figure skating is one of the most popular ice sports at the Winter Olympic Games. The skaters perform several skating skills to express the beauty of the art on ice. Skating involves moving on ice while wearing skate shoes with thin blades; thus, it requires much practice to skate without losing balance. Moreover, figure skating presents dynamic moves, such as jumping, artistically. Therefore, demonstrating figure skating skills is even more difficult to achieve than basic skating, and professional skaters often fall during Winter Olympic performances. We propose a system to demonstrate figure skating motions with a physically simulated human‐like character. We simulate skating motions with non‐holonomic constraints, which make the skate blade glide on the ice surface. It is difficult to obtain reference motions from figure skaters because figure skating motions are very fast and dynamic. Instead of using motion capture data, we use key poses extracted from videos on YouTube and complete reference motions using trajectory optimization. We demonstrate figure skating skills, such as crossover, three‐turn, and even jump. Finally, we use deep reinforcement learning to generate a robust controller for figure skating skills.

Many anatomical factors, such as bone geometry and muscle condition, interact to affect human movements. This work aims to build a comprehensive musculoskeletal model and its control system that reproduces realistic human movements driven by muscle contraction dynamics. The variations in the anatomic model generate a spectrum of human movements ranging from typical to highly stylistic movements. To do so, we discuss scalable and reliable simulation of anatomical features, robust control of under-actuated dynamical systems based on deep reinforcement learning, and modeling of pose-dependent joint limits. The key technical contribution is a scalable, two-level imitation learning algorithm that can deal with a comprehensive full-body musculoskeletal model with 346 muscles. We demonstrate the predictive simulation of dynamic motor skills under anatomical conditions including bone deformity, muscle weakness, contracture, and the use of a prosthesis. We also simulate various pathological gaits and predictively visualize how orthopedic surgeries improve post-operative gaits.

We present a novel and general framework for the design and control of underwater soft-bodied animals. The whole body of an animal consisting of soft tissues is modeled by tetrahedral and triangular FEM meshes. The contraction of muscles embedded in the soft tissues actuates the body and limbs to move. We present a novel muscle excitation model that mimics the anatomy of muscular hydrostats and their muscle excitation patterns. Our deep reinforcement learning algorithm equipped with the muscle excitation model successfully learned the control policy of soft-bodied animals, which can be physically simulated in real-time, controlled interactively, and resilient to external perturbations. We demonstrate the effectiveness of our approach with various simulated animals including octopuses, lampreys, starfishes, stingrays and cuttlefishes. They learn diverse behaviors such as swimming, grasping, and escaping from a bottle. We also implemented a simple user interface system that allows the user to easily create their creatures.

The goal of this research is to create physically simulated biped characters equipped with a rich repertoire of motor skills. The user can control the characters interactively by modulating their control objectives. The characters can interact physically with each other and with the environment. We present a novel network-based algorithm that learns control policies from unorganized, minimally-labeled human motion data. The network architecture for interactive character animation incorporates an RNN-based motion generator into a DRL-based controller for physics simulation and control. The motion generator guides forward dynamics simulation by feeding a sequence of future motion frames to track. The rich future prediction facilitates policy learning from large training data sets. We will demonstrate the effectiveness of our approach with biped characters that learn a variety of dynamic motor skills from large, unorganized data and react to unexpected perturbation beyond the scope of the training data.

Recently, deep reinforcement learning (DRL) has attracted great attention in designing controllers for physics-based characters. Despite the recent success of DRL, the learned controller is viable for a single character. Changes in body size and proportions require learning controllers from scratch. In this paper, we present a new method of learning parametric controllers for body shape variation. A single parametric controller enables us to simulate and control various characters having different heights, weights, and body proportions. The users are allowed to create new characters through body shape parameters, and they can control the characters immediately. Our characters can also change their body shapes on the fly during simulation. The key to the success of our approach includes the adaptive sampling of body shapes that tackles the challenges in learning parametric controllers, which relies on the marginal value function that measures control capabilities of body shapes. We demonstrate parametric controllers for various physically simulated characters such as bipeds, quadrupeds, and underwater animals.

Juggling is a physical skill which consists in keeping one or several objects in continuous motion in the air by tossing and catching it. Jugglers need a high dexterity to control their throws and catches which require speed, accuracy and synchronization. The more balls we juggle with, the more these qualities have to be strong to achieve this performance. This complex skill is good challenge for realistic physical based simulation which could be useful for jugglers to evaluate the feasibility of their tricks. This simulation has to understand the different notations used in juggling and to apply the mathematical theory of juggling to reproduce it. In this paper, we present a deep reinforcement learning method for both tasks catching and throwing, and we combine them to recreate the all juggling process. Our character is able to react accurately and with enough speed and power to juggle with up to 7 balls, even with external forces applied on it.

Simulating believable virtual crowds has been an important research topic in many research fields such as industry films, computer games, urban engineering, and behavioral science. One of the key capabilities agents should have is navigation, which is reaching goals without colliding with other agents or obstacles. The key challenge here is that the environment changes dynamically, where the current decision of an agent can largely affect the state of other agents as well as the agent in the future. Recently, reinforcement learning with deep neural networks has shown remarkable results in sequential decision-making problems. With the power of convolution neural networks, elaborate control with visual sensory inputs has also become possible. In this paper, we present an agent-based deep reinforcement learning approach for navigation, where only a simple reward function enables agents to navigate in various complex scenarios. Our method is also able to do that with a single unified policy for every scenario, where the scenario-specific parameter tuning is unnecessary. We will show the effectiveness of our method through a variety of scenarios and settings.

We present a multi-segment foot model and its control method for the simulation of realistic bipedal behaviors. The ground reaction force is the only source of control for a biped that stands and walks on its feet. The foot is the body part that interacts with the ground and produces appropriate actuation to the body. Foot anatomy features 26 bones and many more muscles that play an important role in weight transmission, balancing posture and assisting ambulation. Previously, the foot model was often simplified into one or two rigid bodies connected by a revolute joint. We propose a new foot model consisting of multiple segments to accurately reproduce human foot shape and its functionality. Based on the new model, we developed a foot pose controller that can reproduce foot postures that are generally not obtained in motion capture data. We demonstrate the validity of our foot model and the effectiveness of our foot controller with a variety of foot motions in a physics-based simulation.

We present an approach that learns to act from raw motion data for interactive character animation. Our motion generator takes a continuous stream of control inputs and generates the character's motion in an online manner. The key insight is modeling rich connections between a multitude of control objectives and a large repertoire of actions. The model is trained using Recurrent Neural Network conditioned to deal with spatiotemporal constraints and structural variabilities in human motion. We also present a new data augmentation method that allows the model to be learned even from a small to moderate amount of training data. The learning process is fully automatic if it learns the motion of a single character, and requires minimal user intervention if it deals with props and interaction between multiple characters.

We propose a framework for simulation and control of the human musculoskeletal system, capable of reproducing realistic animations of dexterous activities with high-level coordination. We present the first controllable system in this class that incorporates volumetric muscle actuators, tightly coupled with the motion controller, in enhancement of line-segment approximations that prior art is overwhelmingly restricted to. The theoretical framework put forth by our methodology computes all the necessary Jacobians for control, even with the drastically increased dimensionality of the state descriptors associated with three-dimensional, volumetric muscles. The direct coupling of volumetric actuators in the controller allows us to model muscular deficiencies that manifest in shape and geometry, in ways that cannot be captured with line-segment approximations. Our controller is coupled with a trajectory optimization framework, and its efficacy is demonstrated in complex motion tasks such as juggling, and weightlifting sequences with variable anatomic parameters and interaction constraints.

Flying creatures in animated films often perform highly dynamic aerobatic maneuvers, which require their extreme of exercise capacity and skillful control. Designing physics-based controllers (a.k.a., control policies) for aerobatic maneuvers is very challenging because dynamic states remain in unstable equilibrium most of the time during aerobatics. Recently, Deep Reinforcement Learning (DRL) has shown its potential in constructing physics-based controllers. In this paper, we present a new concept, Self-Regulated Learning (SRL) , which is combined with DRL to address the aerobatics control problem. The key idea of SRL is to allow the agent to take control over its own learning using an additional self-regulation policy. The policy allows the agent to regulate its goals according to the capability of the current control policy. The control and self-regulation policies are learned jointly along the progress of learning. Self-regulated learning can be viewed as building its own curriculum and seeking compromise on the goals. The effectiveness of our method is demonstrated with physically-simulated creatures performing aerobatic skills of sharp turning, rapid winding, rolling, soaring, and diving.

Imaginary winged creatures in computer animation applications are expected to perform a variety of motor skills in a physically realistic and controllable manner. Designing physics-based controllers for a flying creature is still very challenging particularly when the dynamic model of the creatures is high-dimensional, having many degrees of freedom. In this paper, we present a control method for flying creatures, which are aerodynamically simulated, interactively controllable, and equipped with a variety of motor skills such as soaring, gliding, hovering, and diving. Each motor skill is represented as Deep Neural Networks (DNN) and learned using Deep Q-Learning (DQL). Our control method is example-guided in the sense that it provides the user with direct control over the learning process by allowing the user to specify keyframes of motor skills. Our novel learning algorithm was inspired by evolutionary strategies of Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to improve the convergence rate and the final quality of the control policy. The effectiveness of our Evolutionary DQL method is demonstrated with imaginary winged creatures flying in a physically simulated environment and their motor skills learned automatically from user-provided keyframes.

Abstract The behavioral structure of human movements is imposed by multiple sources, such as rules, regulations, choreography, habits, and emotion. Our goal is to identify the behavioral structure in a specific application domain and create a novel sequence of movements that abide by structure‐building rules. To do so, we exploit the ideas from formal language, such as rewriting rules and grammar parsing, and adapted those ideas to synthesize the three‐dimensional animation of multiple characters. The structured motion synthesis using motion grammars is formulated in two layers. The upper layer is a symbolic description that relates the semantics of each individual's movements and the interaction among them. The lower layer provides spatial and temporal contexts to the animation. Our multi‐level MCMC (Markov Chain Monte Carlo) algorithm deals with the syntax, semantics, and spatiotemporal context of human motion to produce highly‐structured, animated scenes. The power and effectiveness of motion grammars are demonstrated in animating basketball games from drawings on a tactic board. Our system allows the user to position players and draw out tactical plans, which are animated automatically in virtual environments with three‐dimensional, full‐body characters.

Shadow theatre is a genre of performance art in which the actors are only visible as shadows projected on the screen. The goal of this study is to generate animated characters, the shadows of which match a sequence of target silhouettes. This poses several challenges. The motion of multiple characters are carefully coordinated to form a target silhouette on the screen, and each character's pose should be stable, balanced, and plausible. The resulting character animation should be smooth and coherent spatially and temporally. We formulate the problem as nonlinear constrained optimization with objectives, which were designed to generate plausible human motions. Our optimization algorithm was primarily inspired by the heuristic strategies of professional shadow theatre actors. Their know-how was studied and then incorporated into our optimization formulation. We demonstrate the effectiveness of our approach with a variety of target silhouettes and 3D fabrication of the results.

Abstract Markerless human pose recognition using a single‐depth camera plays an important role in interactive graphics applications and user interface design. Recent pose recognition algorithms have adopted machine learning techniques, utilizing a large collection of motion capture data. The effectiveness of the algorithms is greatly influenced by the diversity and variability of training data. We present a new sampling method that resamples a collection of human motion data to improve the pose variability and achieve an arbitrary size and level of density in the space of human poses. The space of human poses is high dimensional, and thus, brute‐force uniform sampling is intractable. We exploit dimensionality reduction and locally stratified sampling to generate either uniform or application specifically biased distributions in the space of human poses. Our algorithm learns to recognize such challenging poses as sitting, kneeling, stretching, and doing yoga using a remarkably small amount of training data. The recognition algorithm can also be steered to maximize its performance for a specific domain of human poses. We demonstrate that our algorithm performs much better than the Kinect software development kit for recognizing challenging acrobatic poses while performing comparably for easy upright standing poses. Copyright © 2015 John Wiley & Sons, Ltd.

Biped controller design pursues two fundamental goals; simulated walking should look human-like and robust against perturbation while maintaining its balance. Normal gait is a pattern of walking that humans normally adopt in undisturbed situations. It has previously been postulated that normal gait is more energy efficient than abnormal or impaired gaits. However, it is not clear whether normal gait is also superior to abnormal gait patterns with respect to other factors, such as stability. Understanding the correlation between gait and stability is an important aspect of biped controller design. We studied this issue in two sets of experiments with human participants and a simulated biped. The experiments evaluated the degree of resilience to external pushes for various gait patterns. We identified four gait factors that affect the balance-recovery capabilities of both human and simulated walking. We found that crouch gait is significantly more stable than normal gait against lateral push. Walking speed and the timing/magnitude of disturbance also affect gait stability. Our work would provide a potential way to compare the performance of biped controllers by normalizing their output gaits and improve their performance by adjusting these decisive factors.

Editing large-scale crowd animation is a daunting task due to the lack of an efficient manipulation method. This paper presents a novel cage-based editing method for large-scale crowd animation. The cage encloses animated characters and supports convenient space/time manipulation methods that were unachievable with previous approaches. The proposed method is based on a combination of cage-based deformation and as-rigid-as-possible deformation with a set of constraints integrated into the system to produce desired results. Our system allows animators to edit existing crowd animations intuitively with real-time performance while maintaining complex interactions between individual characters. Our examples demonstrate how our cage-based user interfaces mitigate the time and effort for the user to manipulate large crowd animation.

We present a biped locomotion controller for humanoid models actuated by more than a hundred Hill-type muscles. The key component of the controller is our novel algorithm that can cope with step-based biped locomotion balancing and the coordination of many nonlinear Hill-type muscles simultaneously. Minimum effort muscle activations are calculated based on muscle contraction dynamics and online quadratic programming. Our controller can faithfully reproduce a variety of realistic biped gaits (e.g., normal walk, quick steps, and fast run) and adapt the gaits to varying conditions (e.g., muscle weakness, tightness, joint dislocation, and external pushes) and goals (e.g., pain reduction and efficiency maximization). We demonstrate the robustness and versatility of our controller with examples that can only be achieved using highly-detailed musculoskeletal models with many muscles.

In many application areas, such as animation for pre-visualizing movie sequences and choreography for dance or other types of performance, only a high-level description of the desired scene is provided as input, either written or verbal. Such sparsity, however, lends itself well to the creative process, as the choreographer, animator or director can be given more choice and control of the final scene. Animating scenes with multi-character interactions can be a particularly complex process, as there are many different constraints to enforce and actions to synchronize. Our novel 'generate-and-rank' approach rapidly and semi-automatically generates data-driven multi-character interaction scenes from high-level graphical descriptions composed of simple clauses and phrases. From a database of captured motions, we generate a multitude of plausible candidate scenes. We then efficiently and intelligently rank these scenes in order to recommend a small but high-quality and diverse selection to the user. This set can then be refined by re-ranking or by generating alternatives to specific interactions. While our approach is applicable to any scenes that depict multi-character interactions, we demonstrate its efficacy for choreographing fighting scenes and evaluate it in terms of performance and the diversity and coverage of the results.

This special section presents expanded versions of three of the best papers from the 11th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA 2012), which was held in Lausanne, Switzerland, from 29-31 July 2012. SCA has established itself as the premier conference dedicated specifically to innovations in the software and technology of computer animation. SCA 2012 received 80 submissions and each submission was reviewed by at least three members of the international program committee. After a thorough online discussion, the 72-member international program committee decided on the 27 full papers and nine short presentation papers accepted for the final program. Out of 27 full papers, the symposium's Best Papers Award Committee selected one best paper, two runner-ups, and four honorable mentions. The selection was informed by the original reviews and the conference presentations. We are delighted to present three out of the six very best papers of SCA 2012 invited for this special section. Each of the invited papers contains a minimum of 30 percent new material and received at least three reviews, including one reviewer not among the original SCA reviewers.

ABSTRACT This paper presents a real‐time performance animation system that reproduces full‐body character animation based on sparse three‐dimensional (3D) motion sensors on a performer. Producing faithful character animation from this setting is a mathematically ill‐posed problem, because input data from the sensors are not sufficient to determine the full degrees of freedom of a character. Given the input data from 3D motion sensors, we select similar poses from a motion database and build an online local model that transforms the low‐dimensional input signal into a high‐dimensional character pose. A regression method based on kernel canonical correlation analysis (CCA) is employed, because it effectively handles a wide variety of motions. Examples show that various human motions are naturally reproduced by the proposed method. Copyright © 2013 John Wiley & Sons, Ltd.

Recent advances in rendering and data-driven animation have enabled the creation of compelling characters with impressive levels of realism. While data-driven techniques can produce animations that are extremely faithful to the original motion, many challenging problems remain because of the high complexity of human motion. A better understanding of the factors that make human motion recognizable and appealing would be of great value in industries where creating a variety of appealing virtual characters with realistic motion is required. To investigate these issues, we captured thirty actors walking, jogging and dancing, and applied their motions to the same virtual character (one each for the males and females). We then conducted a series of perceptual experiments to explore the distinctiveness and attractiveness of these human motions, and whether characteristic motion features transfer across an individual's different gaits. Average faces are perceived to be less distinctive but more attractive, so we explored whether this was also true for body motion. We found that dancing motions were most easily recognized and that distinctiveness in one gait does not predict how recognizable the same actor is when performing a different motion. As hypothesized, average motions were always amongst the least distinctive and most attractive. Furthermore, as 50% of participants in the experiment were Caucasian European and 50% were Asian Korean, we found that the latter were as good as or better at recognizing the motions of the Caucasian actors than their European counterparts, in particular for dancing males, whom they also rated more highly for attractiveness.

We present a physically based controller that simulates the flapping behavior of a bird in flight. We recorded the motion of a dove using marker-based optical motion capture and high-speed video cameras. The bird flight data thus acquired allow us to parameterize natural wingbeat cycles and provide the simulated bird with reference trajectories to track in physics simulation. Our controller simulates articulated rigid bodies of a bird's skeleton and deformable feathers to reproduce the aerodynamics of bird flight. Motion capture from live birds is not as easy as human motion capture because of the lack of cooperation from subjects. Therefore, the flight data we could acquire were limited. We developed a new method to learn wingbeat controllers even from sparse, biased observations of real bird flight. Our simulated bird imitates life-like flapping of a flying bird while actively maintaining its balance. The bird flight is interactively controllable and resilient to external disturbances.

Simulating multiple character interaction is challenging because character actions must be carefully coordinated to align their spatial locations and synchronized with each other. We present an algorithm to create a dense crowd of virtual characters interacting with each other. The interaction may involve physical contacts, such as hand shaking, hugging, and carrying a heavy object collaboratively. We address the problem by collecting deformable motion patches, each of which describes an episode of multiple interacting characters, and tiling them spatially and temporally. The tiling of motion patches generates a seamless simulation of virtual characters interacting with each other in a nontrivial manner. Our tiling algorithm uses a combination of stochastic sampling and deterministic search to address the discrete and continuous aspects of the tiling problem. Our tiling algorithm made it possible to automatically generate highly complex animation of multiple interacting characters. We achieve the level of interaction complexity far beyond the current state of the art that animation techniques could generate, in terms of the diversity of human behaviors and the spatial/temporal density of interpersonal interactions.

Abstract We propose 2D stick figures as a unified medium for visualizing and searching for human motion data. The stick figures can express a wide range or human motion, and they are easy to be drawn by people without any professional training. In our interface, the user can browse overall motion by viewing the stick figure images generated from the database and retrieve them directly by using sketched stick figures as an input query. We started with a preliminary survey to observe how people draw stick figures. Based on the rules observed from the user study, we developed an algorithm converting motion data to a sequence of stick figures. The feature‐based comparison method between the stick figures provides an interactive and progressive search for the users. They assist the user's sketching by showing the current retrieval result at each stroke. We demonstrate the utility of the system with a user study, in which the participants retrieved example motion segments from the database with 102 motion files by using our interface.

In a virtual world, a group of virtual characters can interact with each other, and these characters may leave a group to join another. The interaction among individuals and groups often produces interesting events in a sequence of animation. The goal of this paper is to discover social events involving mutual interactions or group activities in multicharacter animations and automatically plan a smooth camera motion to view interesting events suggested by our system or relevant events specified by a user. Inspired by sociology studies, we borrow the knowledge in Proxemics, social force, and social network analysis to model the dynamic relation among social events and the relation among the participants within each event. By analyzing the variation of relation strength among participants and spatiotemporal correlation among events, we discover salient social events in a motion clip and generate an overview video of these events with smooth camera motion using a simulated annealing optimization method. We tested our approach on different motions performed by multiple characters. Our user study shows that our results are preferred in 66.19 percent of the comparisons with those by the camera control approach without event analysis and are comparable (51.79 percent) to professional results by an artist.

Abstract We present an interactive method that allows animated characters to navigate through cluttered environments. Our characters are equipped with a variety of motion skills to clear obstacles, narrow passages, and highly constrained environment features. Our control method incorporates a behavior model into well‐known, standard path planning algorithms. Our behavior model, calleddeformable motion, consists of a graph of motion capture fragments. The key idea of our approach is to add flexibility on motion fragments such that we can situate them into a cluttered environment via constraint‐based formulation. We demonstrate our deformable motion for realtime interactive navigation and global path planning in highly constrained virtual environments.

Abstract We present a new approach to motion rearrangement that preserves the syntactic structures of an input motion automatically by learning a context‐free grammar from the motion data. For grammatical analysis, we reduce an input motion into a string of terminal symbols by segmenting the motion into a series of subsequences, and then associating a group of similar subsequences with the same symbol. To obtain the most repetitive and precise set of terminals, we search for an optimial segmentation such that a large number of subsequences can be clustered into groups with little error. Once the input motion has been encoded as a string, a grammar induction algorithm is employed to build up a context‐free grammar so that the grammar can reconstruct the original string accurately as well as generate novel strings sharing their syntactic structures with the original string. Given any new strings from the learned grammar, it is straightforward to synthesize motion sequences by replacing each terminal symbol with its associated motion segment, and stitching every motion segment sequentially. We demonstrate the usefulness and flexibility of our approach by learning grammars from a large diversity of human motions, and reproducing their syntactic structures in new motion sequences.

Crowds and groups are a vital element of life, and simulating them in a convincing manner is one of the great challenges in computer graphics and interactive techniques. This course focuses on the problem of efficiently simulating realistic crowd and group behavior for a range of applications, including games and design of spaces. It covers data driven methods, where the characteristics of crowds are simulated based on real world data; evaluation and perceptual issues, and creation of behavioral variety; interactive simulation and control of large scale crowds and traffic for games and other real time applications; and finally a case study of using crowd simulation for design of spaces in the Disney theme parks.

Data-driven animation using motion capture data has become a standard practice in character animation. A number of techniques have been developed to add flexibility on captured human motion data by editing joint trajectories, warping motion paths, blending a family of parameterized motions, splicing motion segments, and adapting motion to new characters and environments. Even with the abundance of motion capture data and the popularity of data-driven animation techniques, programming with motion capture data is still not easy. A single clip of motion data encompasses a lot of heterogeneous information including joint angles, the position and orientation of the skeletal root, their temporal trajectories, and a number of coordinate systems. Due to this complexity, even simple operations on motion data, such as linear interpolation, are rarely described as succinct mathematical equations in articles. This course provides not only a solid mathematical background but also a practical guide to programming with motion capture data. The course will begin with the brief review of affine geometry and coordinate-invariant (conventionally called coordinate-free) geometric programming, which will generalize incrementally to deal with three-dimensional rotations/orientations, the poses of an articulated figure, and full-body motion data. It will lead to identifying a collection of coordinate-invariant operations on full-body motion data and their object-oriented implementation. Finally, we will discuss the practical use of our programming framework in a variety of contexts ranging from data-driven manipulation/interpolation to state-of-the-art biped locomotion control.

Crowd simulation has been an important research field due to its diverse range of applications that include film production, military simulation, and urban planning. A challenging problem is to provide simple yet effective control over captured and simulated crowds to synthesize intended group motions. We present a new method that blends existing crowd data to generate a new crowd animation. The new animation can include an arbitrary number of agents, extends for an arbitrary duration, and yields a natural-looking mixture of the input crowd data. The main benefit of this approach is to create new spatio-temporal crowd behavior in an intuitive and predictable manner. It is accomplished by introducing a morphable crowd model that allows us to encode the formations and individual trajectories in crowd data. Then, its original spatio-temporal behavior can be reconstructed and interpolated at an arbitrary scale using our morphable model.

We present a dynamic controller to physically simulate under-actuated three-dimensional full-body biped locomotion. Our data-driven controller takes motion capture reference data to reproduce realistic human locomotion through realtime physically based simulation. The key idea is modulating the reference trajectory continuously and seamlessly such that even a simple dynamic tracking controller can follow the reference trajectory while maintaining its balance. In our framework, biped control can be facilitated by a large array of existing data-driven animation techniques because our controller can take a stream of reference data generated on-the-fly at runtime. We demonstrate the effectiveness of our approach through examples that allow bipeds to turn, spin, and walk while steering its direction interactively.

We present a physics-based approach to generate 3D biped character animation that can react to dynamical environments in real time. Our approach utilizes an inverted pendulum model to online adjust the desired motion trajectory from the input motion capture data. This online adjustment produces a physically plausible motion trajectory adapted to dynamic environments, which is then used as the desired motion for the motion controllers to track in dynamics simulation. Rather than using Proportional-Derivative controllers whose parameters usually cannot be easily set, our motion tracking adopts a velocity-driven method which computes joint torques based on the desired joint angular velocities. Physically correct full-body motion of the 3D character is computed in dynamics simulation using the computed torques and dynamical model of the character. Our experiments demonstrate that tracking motion capture data with real-time response animation can be achieved easily. In addition, physically plausible motion style editing, automatic motion transition, and motion adaptation to different limb sizes can also be generated without difficulty.

Abstract The standard C/C++ implementation of a spatial partitioning data structure, such as octree and quadtree, is often inefficient in terms of storage requirements particularly when the memory overhead for maintaining parent‐to‐child pointers is significant with respect to the amount of actual data in each tree node. In this work, we present a novel data structure that implements uniform spatial partitioning without storing explicit parent‐to‐child pointer links. Our linkless tree encodes the storage locations of subdivided nodes using perfect hashing while retaining important properties of uniform spatial partitioning trees, such as coarse‐to‐fine hierarchical representation, efficient storage usage, and efficient random accessibility. We demonstrate the performance of our linkless trees using image compression and path planning examples.

Abstract Given a pair of keyframe formations for a group consisting of multiple individuals, we present a spectral‐based approach to smoothly transforming a source group formation into a target formation while respecting the clusters of the involved individuals. The proposed method provides an effective means for controlling the macroscopic spatiotemporal arrangement of individuals for applications such as expressive formations in mass performances and tactical formations in team sports. Our main idea is to formulate this problem as rotation interpolation of the eigenbases for the Laplacian matrices, each of which represents how the individuals are clustered in a given keyframe formation. A stream of time‐varying formations is controlled by editing the underlying adjacency relationships among individuals as well as their spatial positions at each keyframe, and interpolating the keyframe formations while producing plausible collective behaviors over a period of time. An interactive system of editing existing group behaviors in a hierarchical fashion has been implemented to provide flexible formation control of large crowds.

The ability to interactively edit human motion data is essential for character animation. We present a novel motion editing technique that allows the user to manipulate synchronized multiple character motions interactively. Our Laplacian motion editing method formulates the interaction among multiple characters as a collection of linear constraints and enforces the constraints, while the user directly manipulates the motion of characters in both spatial and temporal domains. Various types of manipulation handles are provided to specify absolute/relative spatial location, direction, time, duration, and synchronization of multiple characters. The capability of non-sequential discrete editing is incorporated into our motion editing interfaces, so continuous and discrete editing is performed simultaneously and seamlessly. We demonstrate that the synchronized multiple character motions are synthesized and manipulated at interactive rates using spatiotemporal constraints.

Abstract Human facial gestures often exhibit such natural stochastic variations as how often the eyes blink, how often the eyebrows and the nose twitch, and how the head moves while speaking. The stochastic movements of facial features are key ingredients for generating convincing facial expressions. Although such small variations have been simulated using noise functions in many graphics applications, modulating noise functions to match natural variations induced from the affective states and the personality of characters is difficult and not intuitive. We present a technique for generating subtle expressive facial gestures (facial expressions and head motion) semi‐automatically from motion capture data. Our approach is based on Markov random fields that are simulated in two levels. In the lower level, the coordinated movements of facial features are captured, parameterized, and transferred to synthetic faces using basis shapes. The upper level represents independent stochastic behavior of facial features. The experimental results show that our system generates expressive facial gestures synchronized with input speech.

Animating a crowd of characters is an important problem in computer graphics. The latest techniques enable highly realistic group motions to be produced in feature animation films and video games. However, interactive methods have not emerged yet for editing the existing group motion of multiple characters. We present an approach to editing group motion as a whole while maintaining its neighborhood formation and individual moving trajectories in the original animation as much as possible. The user can deform a group motion by pinning or dragging individuals. Multiple group motions can be stitched or merged to form a longer or larger group motion while avoiding collisions. These editing operations rely on a novel graph structure, in which vertices represent positions of individuals at specific frames and edges encode neighborhood formations and moving trajectories. We employ a shape-manipulation technique to minimize the distortion of relative arrangements among adjacent vertices while editing the graph structure. The usefulness and flexibility of our approach is demonstrated through examples in which the user creates and edits complex crowd animations interactively using a collection of group motion clips.

Physically based simulation of human motions is an important issue in the context of computer animation, robotics and biomechanics. We present a new technique for allowing our physically-simulated planar biped characters to imitate human behaviors. Our contribution is twofold. We developed an optimization method that transforms any (either motion-captured or kinematically synthesized) biped motion into a physically-feasible, balance-maintaining simulated motion. Our optimization method allows us to collect a rich set of training data that contains stylistic, personality-rich human behaviors. Our controller learning algorithm facilitates the creation and composition of robust dynamic controllers that are learned from training data. We demonstrate a planar articulated character that is dynamically simulated in real time, equipped with an integrated repertoire of motor skills, and controlled interactively to perform desired motions.

Abstract Human motion is difficult to create and manipulate because of the high dimensionality and spatiotemporal nature of human motion data. Recently, the use of large collections of captured motion data has added increased realism in character animation. In order to make the synthesis and analysis of motion data tractable, we present a low‐dimensional motion space in which high‐dimensional human motion can be effectively visualized, synthesized, edited, parameterized, and interpolated in both spatial and temporal domains. Our system allows users to create and edit the motion of animated characters in several ways: The user can sketch and edit a curve on low‐dimensional motion space, directly manipulate the character's pose in three‐dimensional object space, or specify key poses to create in‐between motions. Copyright © 2006 John Wiley & Sons, Ltd.

Real time animation of human figures in virtual environments is an important problem in the context of computer games and virtual environments. Recently, the use of large collections of captured motion data has increased realism in character animation. However, assuming that the virtual environment is large and complex, the effort of capturing motion data in a physical environment and adapting them to an extended virtual environment is the bottleneck for achieving interactive character animation and control. We present a new technique for allowing our animated characters to navigate through a large virtual environment, which is constructed using a set of building blocks. The building blocks, called motion patches , can be arbitrarily assembled to create novel environments. Each patch is annotated with motion data, which informs what actions are available for animated characters within the block. The versatility and flexibility of our approach are demonstrated through examples in which multiple characters are animated and controlled at interactive rates in large, complex virtual environments.

Typical high-level directives for locomotion of human-like characters are useful for interactive games and simulations as well as for off-line production animation. In this paper, we present a new scheme for planning natural-looking locomotion of a biped figure to facilitate rapid motion prototyping and task-level motion generation. Given start and goal positions in a virtual environment, our scheme gives a sequence of motions to move from the start to the goal using a set of live-captured motion clips. Based on a novel combination of probabilistic path planning and hierarchical displacement mapping, our scheme consists of three parts: roadmap construction, roadmap search, and motion generation. We randomly sample a set of valid footholds of the biped figure from the environment to construct a directed graph, called a roadmap, that guides the locomotion of the figure. Every edge of the roadmap is associated with a live-captured motion clip. Augmenting the roadmap with a posture transition graph, we traverse it to obtain the sequence of input motion clips and that of target footprints. We finally adapt the motion sequence to the constraints specified by the footprint sequence to generate a desired locomotion.

Real-time control of three-dimensional avatars is an important problem in the context of computer games and virtual environments. Avatar animation and control is difficult, however, because a large repertoire of avatar behaviors must be made available, and the user must be able to select from this set of behaviors, possibly with a low-dimensional input device. One appealing approach to obtaining a rich set of avatar behaviors is to collect an extended, unlabeled sequence of motion data appropriate to the application. In this paper, we show that such a motion database can be preprocessed for flexibility in behavior and efficient search and exploited for real-time avatar control. Flexibility is created by identifying plausible transitions between motion segments, and efficient search through the resulting graph structure is obtained through clustering. Three interface techniques are demonstrated for controlling avatar motion using this data structure: the user selects from a set of available choices, sketches a path through an environment, or acts out a desired motion in front of a video camera. We demonstrate the flexibility of the approach through four different applications and compare the avatar motion to directly recorded human motion.

Capturing live motion has gained considerable attention in computer animation as an important motion generation technique. Canned motion data are comprised of both position and orientation components. Although a great number of signal processing methods are available for manipulating position data, the majority of these methods cannot be generalized easily to orientation data due to the inherent nonlinearity of the orientation space. In this paper, we present a new scheme that enables us to apply a filter mask (or a convolution filter) to orientation data. The key idea is to transform the orientation data into their analogues in a vector space, to apply a filter mask on them, and then to transform the results back to the orientation space. This scheme gives time-domain filters for orientation data that are computationally efficient and satisfy such important properties as coordinate invariance, time invariance and symmetry. Experimental results indicate that our scheme is useful for various purposes, including smoothing and sharpening.

Computer puppetry maps the movements of a performer to an animated character in real-time. In this article, we provide a comprehensive solution to the problem of transferring the observations of the motion capture sensors to an animated character whose size and proportion may be different from the performer's. Our goal is to map as many of the important aspects of the motion to the target character as possible, while meeting the online, real-time demands of computer puppetry. We adopt a Kalman filter scheme that addresses motion capture noise issues in this setting. We provide the notion of dynamic importance of an end-effector that allows us to determine what aspects of the performance must be kept in the resulting motion. We introduce a novel inverse kinematics solver that realizes these important aspects within tight real-time constraints. Our approach is demonstrated by its application to broadcast television performances.

Digital watermarking is one of commonly used solutions for copyright protection. A watermark should be imperceptible and robust to various attacks. In this paper, we address watermarking for motion data. Our watermarking scheme is based on two well‐known ideas, so called multiresolution representation and spread spectrum. We embed a watermark into a motion signal by perturbing large detail coefficients of its multiresolution representation, and extract the watermark by analyzing perturbation of coefficients from a suspected signal. For more effective watermark extraction, we align suspected motion data to the original using dynamic time warping. Our scheme has merits of spread spectrum such as the resilience to common signal processing as well as the robustness to time warping.

This paper presents a technique for adapting existing motion of a human-like character to have the desired features that are specified by a set of constraints. This problem can be typically formulated as a spacetime constraint problem. Our approach combines a hierarchical curve fitting technique with a new inverse kinematics solver. Using the kinematics solver, we can adjust the configuration of an articulated figure to meet the constraints in each frame. Through the fitting technique, the motion displacement of every joint at each constrained frame is interpolated and thus smoothly propagated to frames. We are able to adaptively add motion details to satisfy the constraints within a specified tolerance by adopting a multilevel B-spline representation which also provides a speedup for the interpolation. The performance of our system is further enhanced by the new inverse kinematics solver. We present a closed-form solution to compute the joint angles of a limb linkage. This analytical m...

Motion capturing is widely used for generating realistic motions. A 3D input device captures a sequence of 6 DOF rigid motion samples. The sampled data consists of two components, translation and orientation; the former is represented by a vector in R/sup 3/ and the latter by a rotation matrix in the rotation group, SO(3). Since the sequence of data contains sampling noises, the captured motion is now smooth and wiggles along the moving path. There are well-known fairing algorithms in Euclidean space based on difference geometry. Extending these for the motion data, we present a new fairing algorithm. The new algorithm iteratively minimizes the energy function reflecting the forces and torques, exerted on a moving object, to perform motion fairing.