• We developed a diffusion-based generative framework called SEN, designed for sketch-guided local shape editing. This framework ensures consistency in geometric coherence and preservation. • We introduced an attention fusion module that effectively connects the cross-modal conditions to produce the final shape editing results. • We developed a sketch generator to directly generate sketch images from point clouds to facilitate our SEN training. Modeling complex 3D shapes from 2D sketch images has gained significant attention due to recent advancements in deep generative models. While many efforts have focused on producing geometries that faithfully reflect sketches, there has been limited exploration of sketch-guided local editing. Existing shape generation techniques often modify not only the intended edited areas but also the unedited parts, compromising geometric coherence and fairness among different parts of the shape. This paper introduces a shape editing network (SEN) under the diffusion-based generative framework to specifically enhance consistency in sketch-guided local shape editing. Our core insight is capturing partial geometric structure correlations to achieve consistent editing results. To accomplish this, the SEN is designed to conditionally learn local edits of a partial shape while considering the geometries of the unedited parts and the sketch image. We also devise an attention fusion module that progressively blends and aligns cross-modal conditioning streams between the latent variables of the sketch and point cloud representations. Additionally, by incorporating a guided sampling strategy for diffusion modeling, our SEN allows for more flexible control, even with complex or limited conditioning patterns. Our method significantly improves the geometric coherence of the synthesized parts while preserving the shapes of unedited areas. We demonstrate the advantages of our approach qualitatively and quantitatively across various shape categories, outperforming state-of-the-art methodologies.

We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video and target speech audio, TANGO produces high-fidelity videos with synchronized body gestures. TANGO builds on Gesture Video Reenactment (GVR), which splits and retrieves video clips using a directed graph structure - representing video frames as nodes and valid transitions as edges. We address two key limitations of GVR: audio-motion misalignment and visual artifacts in GAN-generated transition frames. In particular, i) we propose retrieving gestures using latent feature distance to improve cross-modal alignment. To ensure the latent features could effectively model the relationship between speech audio and gesture motion, we implement a hierarchical joint embedding space (AuMoClip); ii) we introduce the diffusion-based model to generate high-quality transition frames. Our diffusion model, Appearance Consistent Interpolation (ACInterp), is built upon AnimateAnyone and includes a reference motion module and homography background flow to preserve appearance consistency between generated and reference videos. By integrating these components into the graph-based retrieval framework, TANGO reliably produces realistic, audio-synchronized videos and outperforms all existing generative and retrieval methods. Our code, pretrained models, and datasets are publicly available at https://github.com/CyberAgentAILab/TANGO.

Speech-driven facial animation aims to synthesize lip-synchronized 3D talking faces following the given speech signal. Prior methods to this task mostly focus on pursuing realism with deterministic systems, yet characterizing the potentially stochastic nature of facial motions has been to date rarely studied. While generative modeling approaches can easily handle the one-to-many mapping by repeatedly drawing samples, ensuring a diverse mode coverage of plausible facial motions on small-scale datasets remains challenging and less explored. In this article, we propose predicting multiple samples conditioned on the same audio signal and then explicitly encouraging sample diversity to address diverse facial animation synthesis. Our core insight is to guide our model to explore the expressive facial latent space with a diversity-promoting loss such that the desired latent codes for diversification can be ideally identified. To this end, building upon the rich facial prior learned with vector-quantized variational auto-encoding mechanism, our model temporally queries multiple stochastic codes which can be flexibly decoded into a diverse yet plausible set of speech-faithful facial motions. To further allow for control over different facial parts during generation, the proposed model is designed to predict different facial portions of interest in a sequential manner, and compose them to eventually form full-face motions. Our paradigm realizes both diverse and controllable facial animation synthesis in a unified formulation. We experimentally demonstrate that our method yields state-of-the-art performance both quantitatively and qualitatively, especially regarding sample diversity.

Abstract Gestural animations in the amusement or entertainment field often require rich expressions; however, it is still challenging to synthesize characteristic gestures automatically. Although style transfer based on a neural network model is a potential solution, existing methods mainly focus on cyclic motions such as gaits and require re‐training in adding new motion styles. Moreover, their per‐pose transformation cannot consider the time‐dependent features, and therefore motion styles of different periods and timings are difficult to be transferred. This limitation is fatal for the gestural motions requiring complicated time alignment due to the variety of exaggerated or intentionally performed behaviors. This study introduces a context‐based style transfer of gestural motions with neural networks to ensure stable conversion even for exaggerated, dynamically complicated gestures. We present a model based on a vision transformer for transferring gestures' content and style features by time‐segmenting them to compose tokens in a latent space. We extend this model to yield the probability of swapping gestures' tokens for style‐transferring. A transformer model is suited to semantically consistent matching among gesture tokens, owing to the correlation with spoken words. The compact architecture of our network model requires only a small number of parameters and computational costs, which is suitable for real‐time applications with an ordinary device. We introduce loss functions provided by the restoration error of identically and cyclically transferred gesture tokens and the similarity losses of content and style evaluated by splicing features inside the transformer. This design of losses allows unsupervised and zero‐shot learning, by which the scalability for motion data is obtained. We comparatively evaluated our style transfer method, mainly focusing on expressive gestures using our dataset captured for various scenarios and styles by introducing new error metrics tailored for gestures. Our experiment showed the superiority of our method in numerical accuracy and stability of style transfer against the existing methods.

A clip of character motion can be adapted to a change in environment or to another character of a different body size via a numerical optimization with several tasks including the objective of movement and physical constraints. Conventional methods, however, lack the design flexibility of such adaptation tasks because of the simple problem formulation. We propose a motion adaptation framework based on a cascaded series of quadratic programs. Our system introduces a layered structure of strictly prioritized tasks, each layer of which comprises arbitrary types of equality and inequality tasks. The cascaded solver identifies the optimal solution in each layer without affecting the fulfillment of the higher layer tasks. The stable computation of the cascaded optimization supports the intuitive design of the spacetime tasks even for novice users. The capability of our method was demonstrated through several experiments of motion adaptation with prioritized inequality tasks, such as environmental adaptation and adaptation of interactive behavior between two characters.

Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future uncertainty in both predictions by introducing latent codes. Unlike previous methods that infer output frames directly, our CNNs predict spatially-smooth intermediate data, i.e., for motion, flow fields for warping, and for appearance, color transfer maps, via self-supervised learning, i.e., without explicitly-provided ground truth. These intermediate data are applied not to each previous output frame, but to the input image only once for each output frame. This design is crucial to alleviate error accumulation in long-term predictions, which is the essential problem in previous recurrent approaches. The output frames can be looped like cinemagraph, and also be controlled directly by specifying latent codes or indirectly via visual annotations. We demonstrate the effectiveness of our method through comparisons with the state-of-the-arts on video prediction as well as appearance manipulation. Resultant videos, codes, and datasets will be available at http://www.cgg.cs.tsukuba.ac.jp/~endo/projects/AnimatingLandscape.

This paper describes a prototype self-training system for tennis forehand shots that allows trainees to practice their motion forms by themselves. Our system uses a motion capture device to record a trainee's motion, and visualizes the differences between the features of the trainee's motion and the correct motion as performed by an expert. This system enables trainees to understand the errors in their motion and how to reduce or eliminate them. In this study, we classified the motion features and corresponding visualization methods using one-dimensional spatial, rotational, and temporal features based on the key sporting poses. We also developed a statistical model for the motion features, allowing the system to assess and prioritize all features of a trainee's motion. This research focuses on the motion of a tennis forehand shot and evaluates our prototype through several user experiments.

Dynamic skin deformation is vital for creating life-like characters, and its real-time computation is in great demand in interactive applications. We propose a practical method to synthesize plausible and dynamic skin deformation based on a helper bone rig. This method builds helper bone controllers for the deformations caused not only by skeleton poses but also secondary dynamics effects. We introduce a state-space model for a discrete time linear time-invariant system that efficiently maps the skeleton motion to the dynamic movement of the helper bones. Optimal transfer of nonlinear, complicated deformations, including the effect of soft-tissue dynamics, is obtained by learning the training sequence consisting of skeleton motions and corresponding skin deformations. Our approximation method for a dynamics model is highly accurate and efficient owing to its low-rank property obtained by a sparsity-oriented nuclear norm optimization. The resulting linear model is simple enough to easily implement in the existing workflows and graphics pipelines. We demonstrate the superior performance of our method compared to conventional dynamic skinning in terms of computational efficiency including LOD controls, stability in interactive controls, and flexible expression in deformations.

Interactive animation systems often use a level-of-detail (LOD) control to reduce the computational cost by eliminating unperceivable details of the scene. Most methods employ a multiresolutional representation of animation and geometrical data, and adaptively change the accuracy level according to the importance of each character. Multilinear analysis provides the efficient representation of multidimensional and multimodal data, including human motion data, based on statistical data correlations. This paper proposes a LOD control method of motion synthesis with a multilinear model. Our method first extracts a small number of principal components of motion samples by analyzing three-mode correlations among joints, time, and samples using high-order singular value decomposition. A new motion is synthesized by interpolating the reduced components using geostatistics, where the prediction accuracy of the resulting motion is controlled by adaptively decreasing the data dimensionality. We introduce a hybrid algorithm to optimize the reduction size and computational time according to the distance from the camera while maintaining visual quality. Our method provides a practical tool for creating an interactive animation of many characters while ensuring accurate and flexible controls at a modest level of computational cost.

Abstract This paper proposes a psychological model for simulating pedestrian behaviors in a crowded space. Our decision‐making scheme controls plausible avoidance behavior depending on the positional relations among surrounding persons, on the basis of a two‐stage personal space and a virtual memory structure as proposed in social psychology. Our system determines pedestrian walking speed with the crowd density to imitate the measured data in urban engineering, and automatically generates plausible motions of the individual pedestrian by composing a locomotion graph with motion capture data. Our approach based on psychology and a variety of actual measurements can increase the accuracy of simulation at both the micro and macro levels. Copyright © 2005 John Wiley & Sons, Ltd.

A common motion interpolation technique for realistic human animation is to blend similar motion samples with weighting functions whose parameters are embedded in an abstract space. Existing methods, however, are insensitive to statistical properties, such as correlations between motions. In addition, they lack the capability to quantitatively evaluate the reliability of synthesized motions. This paper proposes a method that treats motion interpolations as statistical predictions of missing data in an arbitrarily definable parametric space. A practical technique of geostatistics, called universal kriging, is then introduced for statistically estimating the correlations between the dissimilarity of motions and the distance in the parametric space. Our method statistically optimizes interpolation kernels for given parameters at each frame, using a pose distance metric to efficiently analyze the correlation. Motions are accurately predicted for the spatial constraints represented in the parametric space, and they therefore have few undesirable artifacts, if any. This property alleviates the problem of spatial inconsistencies, such as foot-sliding, that are associated with many existing methods. Moreover, numerical estimates for the reliability of predictions enable motions to be adaptively sampled. Since the interpolation kernels are computed with a linear system in real-time, motions can be interactively edited using various spatial controls.

Cracking is a natural phenomenon commonly observed on the surface of such objects as concrete, clay walls, rice fields, pottery, and tree barks. This paper describes a new physically based method for synthesizing 3D cracking patterns in clay solids by incorporating a moisture model. We employ actually measured values of the physical parameters used in the model. In addition, we describe a new method for dealing with a SD freeform object based upon the close-packed structure. Animated synthesis of not only external cracks but also internal cracks of SD freeform solids is possible

We propose a method for creating human movements by imposing positional constraints of end-effectors at multiple key-frame. We introduce hierarchical reinforcement learning for efficiently searching postures at each key-frame among the huge number of possible candidates. The mechanical structures of virtual characters are also hierarchically decomposed so as to suit the learning mechanism, and each hierarchy prepares templates of discretely sampled postures for narrowing down the searching space. Our method automatically generates complicated sequential movements so that the resulting motions optimize dynamic manipulability for enhancing naturalness of realistic human behaviors.

This paper proposes a method for creating human movements by imposing positional constraints of end-effectors at multiple key-frames. We introduce hierarchical reinforcement learning for efficiently searching postures at each key-frame among the huge number of possible candidates. The mechanical structures of virtual characters are also hierarchically decomposed so as to suit the learning mechanism, and each hierarchy prepares templates of discretely sampled postures for narrowing down the searching space. Our method automatically generates complex movements so that the resulting motions are globally optimized for a whole sequence.

This paper describes a new physically based method for synthesizing 3D cracking patterns in clay solids by incorporating a moisture model. We measure temporal transitions of the physical parameters used in the model. Then we describe a new approach to synthesize cracking patterns in a 3D freeform object by its approximate quantized shape and the mesh relation of its original shape. With this new method, an animated synthesis of not only external cracks but also internal cracks of 3D freeform solids is possible.

Abstract This paper proposes a control methodology for human gait with a pattern generator. The pattern generator generates cyclic signals via a couple of mutually inhibited neurons, and drives a proportional derivative controller that supplies joint angles of a virtual human. The state of the pattern generator is entrained by the signal of the controller, and such mutual feedback stabilizes the generation of rhythmic signals for variable conditions. Legs and arms can automatically synchronize their periodical movements without using a central supervisor because the corresponding neural oscillators mutually feed their output signals. Our system generates various gaits in a common mechanism with a small number of parameters, which is well suited for real‐time, interactive and on‐the‐fly controls. Moreover, the movements obtained from motion capture data can be controlled by introducing adjustable non‐linear filters. Copyright © 2002 John Wiley & Sons, Ltd.

We present a control methodology for locomotion using virtual oscillators whose nonlinear dynamics is generated via a couple of mutually inhibited neurons. A controller that rotates a joint is driven by a neural output, and the state of the neural oscillator is entrained by the signal of the controller in return. Such mutual feedback supplies stable controls of rhythmic motions for variable conditions. Controllers are individually assigned to joints of an articulated figure, and the corresponding oscillators mutually supply the neural signals in order to synchronize periodical movements without using a central supervisor. Our neural pattern generator can control various locomotive patterns in a common mechanism with a small number of parameters, and can flexibly and smoothly control the patterns while maintaining global coordination. Such features are suitable for real-time, interactive, and on-the-fly controls of virtual characters.

This paper presents a method of generating polyhedral surfaces by using a diffusion system that calculates the positional and normal vectors on their vertices. The system generates smooth shapes that satisfy the minimum norm property, and can be extended to imitate the shape controls of curvature continuous surfaces with bias and tension parameters. The shape of a surface is determined by the stable state of nonlinear and local calculations between vertices, and is easily controlled by adding constraints on arbitrary vertices. Such bottom‐up calculation of surfaces enhances flexibility in the interactive design of complicated free‐form shapes.