We propose a multi-user Mixed Reality (MR) telepresence system that allows users to interact by seamlessly visualizing remote environments and avatars overlaid onto their local physical space. Building on prior shared-space approaches, our method first aligns overlapping rooms to maximize a shared space –a common area containing matched real and virtual objects where all users can interact. Uniquely, our system extends beyond this shared space by visualizing non-shared spaces, the remaining part of each room, allowing users to inhabit these distinct areas. To address the issue of overlap between non-shared spaces, we dynamically adjust their visibility based on user proximity, using a Voronoi diagram to prioritize subspaces closer to each user. Visualizing the surrounding space of each user conveys spatial context, helping others interpret their behavior within their environment. Visibility is updated in real time as users move, maintaining a coherent sense of spatial awareness. Through a user study, we demonstrate that our system enhances enjoyment, spatial understanding, and presence compared to shared-space-only approaches. Quantitative results further show that our dynamic visibility modulation improves both personal space preservation and space accessibility relative to static methods. Overall, our system provides users with a seamless, dynamically connected, and shared multi-room environment. We provide an system overview and demo video of our work in the supplementary material.

Abstract Motion retargeting is a well‐established technique in computer animation that adapts source motion to fit characters with different sizes, morphologies, or environments. Recent deep learning methods have shown promising results in retargeting character motion. However, retargeting human‐object interactions to new environments, especially when furniture shapes differ significantly, remains a challenging problem. In this work, we propose a novel retargeting framework to address this challenge by combining motion generative models with optimization‐based pose adaptation. Our framework operates in two stages: first, a key pose generator generates the pose of key joints that preserves the interaction state relative to the new furniture; second, final whole‐body pose is determined by accommodating the key joints' poses through optimization. A crucial step in our framework is generating key poses that maintain the interaction state of the source motion. To achieve this, we introduce the Interaction Intensity Weight (IIW) and structural rays, called InterFaceRays, which together capture the interaction intensity between body parts and furniture surfaces. The IIW generator, a trained MoE‐based decoder from the conditional variational autoencoder (cVAE) model, infers IIWs for the target furniture based on the source motion's interaction state. Extensive experiments demonstrate that our framework effectively retargets continuous character motion across diverse furniture configurations, with the IIW generator significantly enhancing key pose consistency. This hybrid approach offers a robust solution for motion retargeting across dissimilar furniture environments.

We introduce ViSA (Virtual Stunt Actors), an interactive animation system designed to create realistic ballistic stunt actions frequently seen in filmmaking and TV production. By providing spatial constraints suitable for the desired stunt scene, our system generates physically plausible motions satisfying the given constraints. The problem is formulated as a deep reinforcement learning task, incorporating a novel state and action spaces, as well as straightforward yet effective rewards for ballistic stunt actions. Users can receive a fast response within several minutes and continue to choreograph complex stunt scenes in an interactive manner. We demonstrate ballistic stunt scenes resembling those in various films and TV dramas, such as traffic accidents, falling down stairs, and falls from buildings. The effectiveness of the technical components and design choices in our system is demonstrated through extensive comparisons, analyses, and ablation studies.

In mixed reality (MR) avatar-mediated telepresence, avatar movement must be adjusted to convey the user's intent in a dissimilar space. This paper presents a novel neural network-based framework designed for translating upper-body gestures, which adjusts virtual avatar movements in dissimilar environments to accurately reflect the user's intended gestures in real-time. Our framework translates a wide range of upper-body gestures, including eye gaze, deictic gestures, free-form gestures, and the transitions between them. A key feature of our framework is its ability to generate natural upper-body gestures for users of different sizes, irrespective of handedness and eye dominance, even though the training is based on data from a single person. Unlike previous methods that require paired motion between users and avatars for training, our framework uses an unpaired approach, significantly reducing training time and allowing for generating a wider variety of motion types. These advantages were made possible by designing two separate networks: the Motion Progression Network, which interprets sparse tracking signals from the user to determine motion progression, and the Upper-body Gesture Network, which autoregressively generates the avatar's pose based on these progressions. We demonstrate the effectiveness of our framework through quantitative comparisons with state-of-the-art methods, qualitative animation results, and a user evaluation in MR telepresence scenarios.

Abstract Full‐body avatar presence is important for immersive social and environmental interactions in digital reality. However, current devices only provide three six degrees of freedom (DOF) poses from the headset and two controllers (i.e. three‐point trackers). Because it is a highly under‐constrained problem, inferring full‐body pose from these inputs is challenging, especially when supporting the full range of body proportions and use cases represented by the general population. In this paper, we propose a deep learning framework, DivaTrack, which outperforms existing methods when applied to diverse body sizes and activities. We augment the sparse three‐point inputs with linear accelerations from Inertial Measurement Units (IMU) to improve foot contact prediction. We then condition the otherwise ambiguous lower‐body pose with the predictions of foot contact and upper‐body pose in a two‐stage model. We further stabilize the inferred full‐body pose in a wide range of configurations by learning to blend predictions that are computed in two reference frames, each of which is designed for different types of motions. We demonstrate the effectiveness of our design on a large dataset that captures 22 subjects performing challenging locomotion for three‐point tracking, including lunges, hula‐hooping, and sitting. As shown in a live demo using the Meta VR headset and Xsens IMUs, our method runs in real‐time while accurately tracking a user's motion when they perform a diverse set of movements.

This paper introduces ELMO, a real-time upsampling motion capture framework designed for a single LiDAR sensor. Modeled as a conditional autoregressive transformer-based upsampling motion generator, ELMO achieves 60 fps motion capture from a 20 fps LiDAR point cloud sequence. The key feature of ELMO is the coupling of the self-attention mechanism with thoughtfully designed embedding modules for motion and point clouds, significantly elevating the motion quality. To facilitate accurate motion capture, we develop a one-time skeleton calibration model capable of predicting user skeleton off-sets from a single-frame point cloud. Additionally, we introduce a novel data augmentation technique utilizing a LiDAR simulator, which enhances global root tracking to improve environmental understanding. To demonstrate the effectiveness of our method, we compare ELMO with state-of-the-art methods in both image-based and point cloud-based motion capture. We further conduct an ablation study to validate our design principles. ELMO's fast inference time makes it well-suited for real-time applications, exemplified in our demo video featuring live streaming and interactive gaming scenarios. Furthermore, we contribute a high-quality LiDAR-mocap synchronized dataset comprising 20 different subjects performing a range of motions, which can serve as a valuable resource for future research. The dataset and evaluation code are available at https://movin3d.github.io/ELMO_SIGASIA2024/

Rapid advances in technology gradually realize immersive mixed-reality (MR) telepresence between distant spaces. This paper presents a novel visual guidance system for avatar-mediated telepresence, directing users to optimal placements that facilitate the clear transfer of gaze and pointing contexts through remote avatars in dissimilar spaces, where the spatial relationship between the remote avatar and the interaction targets may differ from that of the local user. Representing the spatial relationship between the user/avatar and interaction targets with angle-based interaction features, we assign recommendation scores of sampled local placements as their maximum feature similarity with remote placements. These scores are visualized as color-coded 2D sectors to inform the users of better placements for interaction with selected targets. In addition, virtual objects of the remote space are overlapped with the local space for the user to better understand the recommendations. We examine whether the proposed score measure agrees with the actual user perception of the partner's interaction context and find a score threshold for recommendation through user experiments in virtual reality (VR). A subsequent user study in VR investigates the effectiveness and perceptual overload of different combinations of visualizations. Finally, we conduct a user study in an MR telepresence scenario to evaluate the effectiveness of our method in real-world applications.

Avatar-mediated augmented reality telepresence aims to enable distant users to collaborate remotely through avatars. When two spaces involved in telepresence are dissimilar, with different object sizes and arrangements, the avatar movement must be adjusted to convey the user’s intention rather than directly following their motion, which poses a significant challenge. In this paper, we propose a novel neural network-based framework for real-time retargeting of users’ deictic motions (pointing at and touching objects) to virtual avatars in dissimilar environments. Our framework translates the user’s deictic motion, acquired from a sparse set of tracking signals, to the virtual avatar’s deictic motion for a corresponding remote object in real-time. One of the main features of our framework is that a single trained network can generate natural deictic motions for various sizes of users. To this end, our network includes two sub-networks: AngleNet and MotionNet. AngleNet maps the angular state of the user’s motion into a latent representation, which is subsequently converted by MotionNet into the avatar’s pose, considering the user’s scale. We validate the effectiveness of our method in terms of deictic intention preservation and movement naturalness through quantitative comparison with alternative approaches. Additionally, we demonstrate the utility of our approach through several AR telepresence scenarios.

Deep learning-based cloth simulation approaches have potential in achieving real-time simulation of complex cloth by directly learning a mapping from control input to resulting cloth movement, bypassing the need for time-consuming dynamic solving and collision processing. Recent advancements have demonstrated the effectiveness of Graph Neural Networks (GNN) in learning cloth dynamics. However, existing GNN-based models have limitations in predicting scenarios involving complex cloth movement. To overcome this limitation, we propose a novel GNN-based model that incorporates several components, including RNN-based state encoding and physics-informed features. Our model significantly improves the accuracy of cloth dynamics prediction in various scenarios, including those with complex cloth movement driven by control handles. Furthermore, our model demonstrates generalization capabilities for cloth mesh topology and control handle configurations. We validate the effectiveness of our approach through ablation studies and comparisons with a baseline model.

Transforming neutral, characterless input motions to embody the distinct style of a notable character in real time is highly compelling for character animation. This paper introduces MOCHA, a novel online motion characterization framework that transfers both motion styles and body proportions from a target character to an input source motion. MOCHA begins by encoding the input motion into a motion feature that structures the body part topology and captures motion dependencies for effective characterization. Central to our framework is the Neural Context Matcher, which generates a motion feature for the target character with the most similar context to the input motion feature. The conditioned autoregressive model of the Neural Context Matcher can produce temporally coherent character features in each time frame. To generate the final characterized pose, our Characterizer network incorporates the characteristic aspects of the target motion feature into the input motion feature while preserving its context. This is achieved through a transformer model that introduces the adaptive instance normalization and context mapping-based cross-attention, effectively injecting the character feature into the source feature. We validate the performance of our framework through comparisons with prior work and an ablation study. Our framework can easily accommodate various applications, including characterization with only sparse input and real-time characterization. Additionally, we contribute a high-quality motion dataset comprising six different characters performing a range of motions, which can serve as a valuable resource for future research.

Abstract Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full‐body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data‐driven generative method for real‐time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from LiDAR. As a central factor for high‐accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state‐of‐the‐art methods. Additionally, we implement a real‐time application to showcase our method in real‐world scenarios. MOVIN dataset is available at https://movin3d.github.io/movin_pg2023/https://movin3d.github.io/movin_pg2023/">https://movin3d.github.io/movin_pg2023/ .

Abstract We present DAFNet, a novel data‐driven framework capable of generating various actions for indoor environment interactions. By taking desired root and upper‐body poses as control inputs, DAFNet generates whole‐body poses suitable for furniture of various shapes and combinations. To enable the generation of diverse actions, we introduce an action predictor that automatically infers the probabilities of individual action types based on the control input and environment. The action predictor is learned in an unsupervised manner by training Gaussian Mixture Variational Autoencoder (GMVAE). Additionally, we propose a two‐part normalizing flow‐based pose generator that sequentially generates upper and lower body poses. This two‐part model improves motion quality and the accuracy of satisfying conditions over a single model generating the whole body. Our experiments show that DAFNet can create continuous character motion for indoor scene scenarios, and both qualitative and quantitative evaluations demonstrate the effectiveness of our framework. We propose DAFNet, a novel data‐driven framework that can generate various actions for indoor environment interactions. Given the desired root and upper‐body pose as control inputs, DAFNet generates whole‐body poses for a character appropriate for furniture of various shapes and combinations. image

Laban Movement Analysis (LMA) and its Effort element provide a conceptual framework through which we can observe, describe, and interpret the intention of movement. Effort attributes provide a link between how people move and how their movement communicates to others. It is crucial to investigate the perceptual characteristics of Effort to validate whether it can serve as an effective framework to support a wide range of applications in animation and robotics that require a system for creating or perceiving expressive variation in motion. To this end, we first constructed an Effort motion database of short video clips of five different motions: walk, sit down, pass, put, wave performed in eight ways corresponding to the extremes of the Effort elements. We then performed a perceptual evaluation to examine the perceptual consistency and perceived associations among Effort elements: Space (Indirect/Direct), Time (Sustained/Sudden), Weight (Light/Strong), and Flow (Free/Bound) that appeared in the motion stimuli. The results of the perceptual consistency evaluation indicate that although the observers do not perceive the LMA Effort element 100% as intended, true response rates of seven Effort elements are higher than false response rates except for light Effort. The perceptual consistency results showed varying tendencies by motion. The perceptual association between LMA Effort elements showed that a single LMA Effort element tends to co-occur with the elements of other factors, showing significant correlation with one or two factors (e.g., indirect and free, light and free).

This article presents Motion Puzzle, a novel motion style transfer network that advances the state-of-the-art in several important respects. The Motion Puzzle is the first that can control the motion style of individual body parts, allowing for local style editing and significantly increasing the range of stylized motions. Designed to keep the human’s kinematic structure, our framework extracts style features from multiple style motions for different body parts and transfers them locally to the target body parts. Another major advantage is that it can transfer both global and local traits of motion style by integrating the adaptive instance normalization and attention modules while keeping the skeleton topology. Thus, it can capture styles exhibited by dynamic movements, such as flapping and staggering, significantly better than previous work. In addition, our framework allows for arbitrary motion style transfer without datasets with style labeling or motion pairing, making many publicly available motion datasets available for training. Our framework can be easily integrated with motion generation frameworks to create many applications, such as real-time motion transfer. We demonstrate the advantages of our framework with a number of examples and comparisons with previous work.

The fields of SocialVR, performance capture, and virtual try-on are often faced with a need to faithfully reproduce real garments in the virtual world. One critical task is the disentanglement of the intrinsic garment shape from deformations due to fabric properties, physical forces, and contact with the body. We propose to use a garment sewing pattern, a realistic and compact garment descriptor, to facilitate the intrinsic garment shape estimation. Another major challenge is a high diversity of shapes and designs in the domain. The most common approach for Deep Learning on 3D garments is to build specialized models for individual garments or garment types. We argue that building a unified model for various garment designs has the benefit of generalization to novel garment types, hence covering a larger design domain than individual models would. We introduce NeuralTailor, a novel architecture based on point-level attention for set regression with variable cardinality, and apply it to the task of reconstructing 2D garment sewing patterns from the 3D point cloud garment models. Our experiments show that NeuralTailor successfully reconstructs sewing patterns and generalizes to garment types with pattern topologies unseen during training.

Rapidly developing technologies are realizing a 3D telepresence, in which geographically separated users can interact with each other through their virtual avatars. In this article, we present novel methods to determine the avatar's position in an indoor space to preserve the semantics of the user's position in a dissimilar indoor space with different space configurations and furniture layouts. To this end, we first perform a user survey on the preferred avatar placements for various indoor configurations and user placements, and identify a set of related attributes, including interpersonal relation, visual attention, pose, and spatial characteristics, and quantify these attributes with a set of features. By using the obtained dataset and identified features, we train a neural network that predicts the similarity between two placements. Next, we develop an avatar placement method that preserves the semantics of the placement of the remote user in a different space as much as possible. We show the effectiveness of our methods by implementing a prototype AR-based telepresence system and user evaluations.

Abstract The acquisition of highly detailed static 3D scan data for people in clothing is becoming widely available. Since 3D scan data is given as a single mesh without semantic separation, in order to animate the data, it is necessary to model shape and deformation behaviour of individual body and garment parts. This paper presents a new method for generating simulation‐ready garment models from 3D static scan data of clothed humans. A key contribution of our method is a novel approach to segmenting garments by finding optimal boundaries between the skin and garment. Our boundary‐based garment segmentation method allows for stable and smooth separation of garments by using an implicit representation of the boundary and its optimization strategy. In addition, we present a novel framework to construct a 2D pattern from the segmented garment and place it around the body for a draping simulation. The effectiveness of our method is validated by generating garment patterns for a number of scan data.

Abstract With the popularization of games and VR/AR devices, there is a growing need for capturing human motion with a sparse set of tracking data. In this paper, we introduce a deep neural network (DNN) based method for real‐time prediction of the lower‐body pose only from the tracking signals of the upper‐body joints. Specifically, our Gated Recurrent Unit (GRU)‐based recurrent architecture predicts the lower‐body pose and feet contact states from a past sequence of tracking signals of the head, hands, and pelvis. A major feature of our method is that the input signal is represented by the velocity of tracking signals. We show that the velocity representation better models the correlation between the upper‐body and lower‐body motions and increases the robustness against the diverse scales and proportions of the user body than position‐orientation representations. In addition, to remove foot‐skating and floating artifacts, our network predicts feet contact state, which is used to post‐process the lower‐body pose with inverse kinematics to preserve the contact. Our network is lightweight so as to run in real‐time applications. We show the effectiveness of our method through several quantitative evaluations against other architectures and input representations with respect to wild tracking data obtained from commercial VR devices.

This special issue contains 24 full papers selected from the Computer Animation and Social Agents 2021 Conference (CASA2021). This conference was founded by the Computer Graphics Society (CGS) in 1988 in Geneva and is the oldest conference on Computer Animation in the world. It has been held in many countries around the world and in recent years in Beijing, China (2018), Paris, France (2019), Bournemouth, UK (2020), and this year in Ottawa, Canada. The two last conferences have been organized virtually due to the Covid-19 pandemy.

Telepresence avatars enable users in different environments to interact with each other. In order to increase the effectiveness of these interactions, however, the movements of avatars must be adjusted accordingly to account for differences between user environments. For instance, if a user moves from one point to another in one environment, the avatar’s locomotion speed must be adjusted to move to the corresponding target point in another environment at the same time. Several locomotion styles can be used to achieve this speed change. This paper investigates how different avatar locomotion styles (speed, stride, and glide), body visibility levels (full body and head-to-knee), and views (front views and side views) influence human perceptions of the naturalness of motion, similarity to the user’s locomotion, and the degree of preserving the user’s intention. Our results indicate that 1) speed and stride styles are perceived as more natural than the glide style, while the glide style is more intention-preserving than the others, 2) a greater locomotion speed of the avatar is perceived as more natural, similar, and intention-preserving than slower motion, 3) the perception of naturalness has the greatest impact on people’s preferences for locomotion styles, and that 4) head-to-knee body visibility may enhance the perception of naturalness for the glide style.

Abstract This paper presents a novel recurrent neural network‐based method to construct a latent motion manifold that can represent a wide range of human motions in a long sequence. We introduce several new components to increase the spatial and temporal coverage in motion space while retaining the details of motion capture data. These include new regularization terms for the motion manifold, combination of two complementary decoders for predicting joint rotations and joint velocities and the addition of the forward kinematics layer to consider both joint rotation and position errors. In addition, we propose a set of loss terms that improve the overall quality of the motion manifold from various aspects, such as the capability of reconstructing not only the motion but also the latent manifold vector, and the naturalness of the motion through adversarial loss. These components contribute to creating compact and versatile motion manifold that allows for creating new motions by performing random sampling and algebraic operations, such as interpolation and analogy, in the latent motion manifold.

When designing comfort and usability in products, designers need to evaluate aspects ranging from anthropometrics to use scenarios. Therefore, virtual and poseable mannequins are employed as a reference in early-stage tools and for evaluation in the later stages. However, tools to intuitively interact with virtual humans are lacking. In this paper, we introduceSmartManikin, a mannequin with agency that responds to high-level commands and to real-time design changes. We first captured human poses with respect to desk configurations, identified key features of the pose and trained regression functions to estimate the optimal features at a given desk setup. The SmartManikin's pose is generated by the predicted features as well as by using forward and inverse kinematics. We present our design, implementation, and an evaluation with expert designers. The results revealed that SmartManikin enhances the design experience by providing feedback concerning comfort and health in real time.

Abstract As the number of models for 3D indoor scenes are increasing rapidly, methods for generating the lighting layout have also become increasingly important. This paper presents a novel method that creates optimal placements and intensities of a set of lights in indoor scenes. Our method is characterized by designing the objective functions for the optimization based on the lighting guidelines used in the interior design field. Specifically, to apply major elements of the lighting guideline, we identify three criteria, namely the structure, function, and aesthetics, that are suitable for the virtual space and quantify them through a set of objective terms: pairwise relation, hierarchy, circulation, illuminance, and collision. Given an indoor scene with properly arranged furniture as input, our method combines the procedural and optimization‐based approaches to generate lighting layouts appropriate to the geometric and functional characteristics of the input scene. The effectiveness of our method is demonstrated with an ablation study of cost terms for the optimization and a user study for perceptual evaluation.

When motion capture data is applied to virtual characters, the applied motion often exhibits geometric and physical errors, which necessitates a cumbersome refinement process. We present a novel framework to efficiently obtain a corrected motion as well as its supporting contact information from multi-contact motion capture data. To this end, first, we present a projective dynamics-based method for optimizing character motions. By carefully defining objective functions and constraints using differential representation of motions, we develop a highly efficient motion optimizer that can create geometrically and dynamically adjusted motions given reference motion data and contact information. Second, we develop a contact optimizer that finds a set of contacts that allows the motion optimizer to generate a motion that best follows the reference motion under dynamic and geometric constraints. This is achieved by iteratively improving the hypothesis on the best set of contacts by getting feedback from the motion optimizer. We demonstrate that our method significantly improves the naturalness of a wide range of motion capture data, from walking to rolling.

Abstract Applying motion‐capture data to multi‐person interaction between virtual characters is challenging because one needs to preserve the interaction semantics while also satisfying the general requirements of motion retargeting, such as preventing penetration and preserving naturalness. An efficient means of representing interaction semantics is by defining the spatial relationships between the body parts of characters. However, existing methods consider only the character skeleton and thus are not suitable for capturing skin‐level spatial relationships. This paper proposes a novel method for retargeting interaction motions with respect to character skins. Specifically, we introduce the aura mesh, which is a volumetric mesh that surrounds a character's skin. The spatial relationships between two characters are computed from the overlap of the skin mesh of one character and the aura mesh of the other, and then the interaction motion retargeting is achieved by preserving the spatial relationships as much as possible while satisfying other constraints. We show the effectiveness of our method through a number of experiments.

Abstract As the deformation behaviors of hair strands vary greatly depending on the hairstyle, the computational cost and accuracy of hair movement simulations can be significantly improved by applying simulation methods specific to a certain style. This paper makes two contributions with regard to the simulation of various hair styles. First, we propose a novel method to reconstruct simulatable hair strands from hair meshes created by artists. Manually created hair meshes consist of numerous mesh patches, and the strand reconstruction process is challenged by the absence of connectivity information among the patches for the same strand and the omission of hidden parts of strands due to the manual creation process. To this end, we develop a two‐stage spectral clustering method for estimating the degree of connectivity among patches and a strand‐growing method that preserves hairstyles. Next, we develop a hairstyle classification method for style‐specific simulations. In particular, we propose a set of features for efficient classifications and show that classifiers trained with the proposed features have higher accuracy than those trained with naive features. Our method applies efficient simulation methods according to the hairstyle without specific user input, and thus is favorable for real‐time simulation.

Despite the recent advances in automatic methods for computing skinning weights, manual intervention is still indispensable to produce high-quality character deformation. However, current modeling software does not provide efficient tools for the manual definition of skinning weights. Widely used paint-based interfaces give users high degrees of freedom, but at the expense of significant efforts and time. This article presents a novel interface for editing skinning weights based on splines, which represent the isolines of skinning weights on a mesh. When a user drags a small number of spline anchor points, our method updates the shape of the isolines and smoothly interpolates or propagates the weights while respecting the given iso-value on the spline. We introduce several techniques to enable the interface to run in real-time and propose a particular combination of functions that generates appropriate skinning weight over the surface. Users can create skinning weights from scratch by using our method. In addition, we present the spline and the gradient fitting methods that closely approximate initial given weights, so that a user can modify the weights with our spline interface. We show the effectiveness of our spline-based interface through a number of test cases.

Abstract Detecting anatomical landmarks on various human models with dynamic poses remains an important and challenging problem in computer graphics research. We present a novel framework that consists of two‐level regressors for finding correlations between human shapes and landmark positions in both body part and holistic scales. To this end, we first develop pose invariant coordinates of landmarks that represent both local and global shape features by using the pose invariant local shape descriptors and their spatial relationships. Our body part‐level regression deals with the shape features from only those body parts that correspond to a certain landmark. In order to do this, we develop a method that identifies such body parts per landmark, by using geometric shape dictionary obtained through the bag of features method. Our method is nearly automatic, as it requires human assistance only once to differentiate the left and right sides. The method also shows the prediction accuracy comparable to or better than those of existing methods, with a test data set containing a large variation of human shapes and poses.

Abstract In avatar‐mediated telepresence, remote users find it difficult to engage and maintain contact, such as a handshake, with each other without a haptic device. We address the problem of adjusting an avatar's pose to promote multifinger contact interaction between remote users. To this end, we first construct a contact point database for nine types of contact interactions between hands through contact experiments with human subjects. We then develop an optimization‐based framework to compute the avatar's pose that realizes the desired contact learned from the experiment while maintaining the naturalness of the hand pose. We show that our method improves the quality of hand interaction for the predefined set of social interactions.

Multi-contact locomotion that uses both the hands and feet in a complex environment remains a challenging problem in computer animation. To address this problem, we present a contact graph, which is a motion graph augmented by learned feasibility predictors, namely contact spaces and an occupancy estimator, for a motion clip in each graph node. By estimating the feasibilities of candidate contact points that can be reached by modifying a motion clip, the predictors allow us to find contact points that are likely to be valid and natural before attempting to generate the actual motion for the contact points. The contact graph thus enables the efficient generation of multi-contact motion in two steps: planning contact points to the goal and then generating the whole-body motion. We demonstrate the effectiveness of our method by creating several climbing motions in complex and cluttered environments by using only a small number of motion samples.

Data driven models of human poses and soft-tissue deformations can produce very realistic results, but they only model the visible surface of the human body and cannot create skin deformation due to interactions with the environment. Physical simulations can generalize to external forces, but their parameters are difficult to control. In this paper, we present a layered volumetric human body model learned from data. Our model is composed of a data-driven inner layer and a physics-based external layer. The inner layer is driven with a volumetric statistical body model (VSMPL). The soft tissue layer consists of a tetrahedral mesh that is driven using the finite element method (FEM). Model parameters, namely the segmentation of the body into layers and the soft tissue elasticity, are learned directly from 4D registrations of humans exhibiting soft tissue deformations. The learned two layer model is a realistic full-body avatar that generalizes to novel motions and external forces. Experiments show that the resulting avatars produce realistic results on held out sequences and react to external forces. Moreover, the model supports the retargeting of physical properties from one avatar when they share the same topology.

We present an avatar animation technique for a telepresence system that allows for the hand contact, especially handshaking, between remote users. The key idea is that, while the avatar follows the remote user's motion normally, it modifies the motion to create and maintain hand contact with the local user when the two users try to engage hand contact. To this end, we develop the support vector machine (SVM)-based classifiers to recognize the users' intention for contact interaction, and online motion generation method to create realistic image sequence of an avatar to realize the continuous contact with the user. A user study has been conducted to verify the effect of our method on the social telepresence.

Abstract 3D objects of the same kind often have different topologies, and finding correspondence between them is important for operations such as morphing, attribute transfer, and shape matching. This paper presents a novel method to find the surface correspondence between topologically different surfaces. The method is characterized by deforming the source polygonal mesh to match the target mesh by using the intermediate implicit surfaces, and by performing a topological surgery at the appropriate locations on the mesh. In particular, we propose a mathematically well‐defined way to detect the topology change of surface by finding the non‐degenerate saddle points of the velocity fields that tracks implicit surfaces. We show the effectiveness and possible applications of the proposed method through several experiments.

In augmented reality (AR) applications, a virtual avatar serves as a useful medium to represent a human in a different place. This paper deals with the problem of retargeting a human motion to an avatar. In particular, we present a novel method that retargets a human motion with respect to an object to that of an avatar with respect to a different object of a similar shape. To achieve this, we developed a spatial map that defines the correspondences between any points in the 3D spaces around the respective objects. The key advantage of the spatial map is that it identifies the desired locations of the avatar's body parts for any input motion of a human. Once the spatial map is created offline, the motion retargeting can be performed in real-time. The retargeted motion preserves important features of the original motion such as the human pose and the spatial relation with the object. We report the results of a number of experiments that demonstrate the effectiveness of the proposed method.

Social touch such as a handshake increases the sense of coexistence and closeness between remote users in a social telepresence environment, but creating such coordinated contact movements with a distant person is extremely difficult if given only visual feedback, without haptic feedback. This paper presents a method to enable hand-contact interaction between remote users in an avatar-mediated telepresence environment. The key approach is, while the avatar directly follows its owner's motion in normal conditions, it adjusts the pose to maintain contact with the other user when the two users attempt to make contact interaction. To this end, we develop classifiers to recognize the users' intention for the contact interaction. The contact classifier identifies whether the users try to initiate contact when they are not in contact, and the separation classifier identifies whether the two in contact attempt to break contact. The classifiers are trained based on a set of geometric distance features. During the contact phase, inverse kinematics is solved to determine the pose of the avatar's arm so as to initiate and maintain natural contact with the other user's hand. Our system is unique in that two remote users can perform real time hand contact interaction in a social telepresence environment.

Rigging is a core element in the process of bringing a 3D character to life. The rig defines and delimits the motions of the character and provides an interface for an animator with which to interact with the 3D character. The quality of the rig has a key impact on the expressiveness of the character. Creating a usable, rich, production ready rig is a laborious task requiring direct intervention by a trained professional because the goal is difficult to achieve with fully automatic methods. We propose a semi-automatic rigging editing framework which eases the need for manual intervention while maintaining an important degree of control over the final rig. Starting by automatically generated base rig, we provide interactive operations which efficiently configure the skeleton structure and mesh skinning.

Abstract We present a novel method to generate a virtual character's multi‐contact poses adaptive to the various shapes of the environment. Given the user‐specified center of mass (CoM) position and direction as inputs, our method finds the potential contacts for the character in the surrounding geometry of the environment and generates a set of stable poses that are contact‐rich. Major contributions of the work are in efficiently finding admissible support points for the target environment by precomputing candidate support points from a human pose database, and in automatically generating interactive poses that can maintain stable equilibrium. We develop the concept of support complexity to scale the set of precomputed support points by the geometric complexity of the environment. We demonstrate the effectiveness of our method by creating contact poses for various test cases of environments.

We address the challenging problem of controlling a complex biomechanical model of the human body to synthesize realistic swimming animation. Our human model includes all of the relevant articular bones and muscles, including 103 bones (163 articular degrees of freedom) plus a total of 823 muscle actuators embedded in a finite element model of the musculotendinous soft tissues of the body that produces realistic deformations. To coordinate the numerous muscle actuators in order to produce natural swimming movements, we develop a biomimetically motivated motor control system based on Central Pattern Generators (CPGs), which learns to produce activation signals that drive the numerous muscle actuators.

In video surveillance and monitoring systems based on wireless mesh networks, large volumes of multimedia data with different priorities and time demands are generated by video cameras and wireless sensors. Hence, one of the key issues is how to alleviate network congestion for highly reliable transfer of multimedia data towards gateways. This paper proposes a solution for utilizing multiple gateways to divert data streams and alleviate traffic congestion near a specific gateway. With a new routing protocol, named “Multi-Gateway Routing with Congestion Avoidance (MGR-CA)”, network congestion can be predicted in a distributed manner and amounts of data traffic transmitted to the congested path are redirected to an alternative gateway based on the severity of congestion. Preliminary testbed experiments are made to show the performance of our scheme.

Geocasting is an efficient mechanism for disseminating messages towards a specific geographical region. Since its packet forwarding is based on the location information of nodes, it does not require any periodic or on-demand path updates. Especially, geocasting is suitable for several applications in tactical ad hoc networks such as alarms for chemical and missile attacks, guerrilla detection or a local weathercast. However, traditional geocast protocols proposed for pure mobile ad hoc networks are inadequate to meet high demands of reliability in military applications. In this study, the authors propose even more efficient and reliable dual-path geocasting protocol which utilises two independent paths and novel acknowledgement mechanisms to improve the chances of successful message delivery. The authors investigate how to locate this dual path to achieve the best performance by means of destination points. The authors’ comprehensive simulation study using ns-2 shows that the proposed scheme results in high delivery ratio with low packet overhead and latency.

We present a novel momentum-based method for maintaining balance of humanoid robots. By controlling the desired ground reaction force (GRF) and center of pressure (CoP) at each support foot, our method can naturally deal with non-level and non-stationary ground at each foot-ground contact, as well as different frictional properties. We do not make use of the net GRF and CoP which may be difficult or impossible to compute for non-level grounds. Our method minimizes the ankle torques during double support. We show the effectiveness of this new balance control method by simulating various experiments with a humanoid robot including maintaining balance when two feet are on separate moving supports with different inclinations and velocities.

We introduce a comprehensive biomechanical model of the human upper body. Our model confronts the combined challenge of modeling and controlling more or less all of the relevant articular bones and muscles, as well as simulating the physics-based deformations of the soft tissues. Its dynamic skeleton comprises 68 bones with 147 jointed degrees of freedom, including those of each vertebra and most of the ribs. To be properly actuated and controlled, the skeletal submodel requires comparable attention to detail with respect to muscle modeling. We incorporate 814 muscles, each of which is modeled as a piecewise uniaxial Hill-type force actuator. To simulate biomechanically-realistic flesh deformations, we also develop a coupled finite element model with the appropriate constitutive behavior, in which are embedded the detailed 3D anatomical geometries of the hard and soft tissues. Finally, we develop an associated physics-based animation controller that computes the muscle activation signals necessary to drive the elaborate musculoskeletal system in accordance with a sequence of target poses specified by an animator.

Spline joints are a novel class of joints that can model general scleronomic constraints for multibody dynamics based on the minimal-coordinates formulation. The main idea is to introduce spline curves and surfaces in the modeling of joints: We model 1-DOF joints using splines on SE(3), and construct multi-DOF joints as the product of exponentials of splines in Euclidean space. We present efficient recursive algorithms to compute the derivatives of the spline joint, as well as geometric algorithms to determine optimal parameters in order to achieve the desired joint motion. Our spline joints can be used to create interesting new simulated mechanisms for computer animation and they can more accurately model complex biomechanical joints such as the knee and shoulder.

A number of conceptually simple but behavior-rich "inverted pendulum" humanoid models have greatly enhanced the understanding and analytical insight of humanoid dynamics. However, these models do not incorporate the robot's angular momentum properties, a critical component of its dynamics. We introduce the reaction mass pendulum (RMP) model, a 3D generalization of the better-known reaction wheel pendulum. The RMP model augments the existing models by compactly capturing the robot's centroidal momenta through its composite rigid body (CRB) inertia. This model provides additional analytical insights into legged robot dynamics, especially for motions involving dominant rotation, and leads to a simpler class of control laws. In this paper we show how a humanoid robot of general geometry and dynamics can be mapped into its equivalent RMP model. A movement is subsequently mapped to the time evolution of the RMP. We also show how an "inertia shaping" control law can be designed based on the RMP.

This paper considers the problem of geocasting in mobile ad hoc networks. Geocasting, a variation on the notion of multicasting, is a mechanism to deliver messages of interest to all nodes within a certain geographical target region. Although several geocasting protocols have already been proposed for mobile ad hoc networks, with the goal of achieving an efficient message delivery, most of these algorithms consider a "single" target region only and therefore multiple transmissions should be initiated separately by the message source when more than one target regions need to receive the same geocast messages. This causes significant performance degradation, especially as the number of geocast regions increase. To solve this problem, we propose a novel scheme driven by geometry, named GGP (geometry-driven geocasting protocol). In this scheme, the geometric concept of "Fermat point" is utilized to determine the optimal junction point among multiple geocast regions from the source node, and hence to reduce the overhead of message delivery, while maintaining a high delivery ratio

Unlike the human face, the neck has been largely overlooked in the computer graphics literature, this despite its complex anatomical structure and the important role that it plays in supporting the head in balance while generating the controlled head movements that are essential to so many aspects of human behavior. This paper makes two major contributions. First, we introduce a biomechanical model of the human head-neck system. Emulating the relevant anatomy, our model is characterized by appropriate kinematic redundancy (7 cervical vertebrae coupled by 3-DOF joints) and muscle actuator redundancy (72 neck muscles arranged in 3 muscle layers). This anatomically consistent biomechanical model confronts us with a challenging motor control problem, even for the relatively simple task of balancing the mass of the head in gravity atop the cervical spine. Hence, our second contribution is a novel neuromuscular control model for human head animation that emulates the relevant biological motor control mechanisms. Incorporating low-level reflex and high-level voluntary sub-controllers, our hierarchical controller provides input motor signals to the numerous muscle actuators. In addition to head pose and movement, it controls the tone of mutually opposed neck muscles to regulate the stiffness of the head-neck multibody system. Employing machine learning techniques, the neural networks within our neuromuscular controller are trained offline to efficiently generate the online pose and tone control signals necessary to synthesize a variety of autonomous movements for the behavioral animation of the human head and face.

A nonlinearity compensated smooth frame insertion (NCSFI) method is proposed to reduce motion-blur caused by hold-type display in LCD. Firstly, each original frame is duplicated into two frames. Then in the first frame, spatial high frequency is removed and in the second frame, the same amount of high frequency is enhanced. Finally, look-up table based nonlinearity compensation is applied to the two new frames to compensate the nonlinearity between gray level and panel luminance. Experiments on real-time display system show that the proposed NCSFI method can reduce motion-blur significantly without luminance distortion

This paper describes Newton and quasi-Newton optimization algorithms for dynamics-based robot movement generation. The robots that we consider are modeled as rigid multibody systems containing multiple closed loops, active and passive joints, and redundant actuators and sensors. While one can, in principle, always derive in analytic form the equations of motion for such systems, the ensuing complexity, both numeric and symbolic, of the equations makes classical optimization-based movement-generation schemes impractical for all but the simplest of systems. In particular, numerically approximating the gradient and Hessian often leads to ill-conditioning and poor convergence behavior. We show in this paper that, by extending (to the general class of systems described above) a Lie theoretic formulation of the equations of motion originally developed for serial chains, it is possible to recursively evaluate the dynamic equations, the analytic gradient, and even the Hessian for a number of physically plausible objective functions. We show through several case studies that, with exact gradient and Hessian information, descent-based optimization methods can be forged into an effective and reliable tool for generating physically natural robot movements.

Motion compensated interlaced-to-progressive conversion (MC-IPC) methods employ implicit decision on whether to use the estimated motion vectors in order to prevent spurious errors due to poorly estimated motion vectors. This often limits the performance of the MC-IPC. We present an IPC method based on the pattern matching assisted motion estimation and motion vector histogram analysis. The proposed method employs an explicit decision on whether to use the estimated motion vectors. Extensive experiments show that high quality IPC can be achieved without spurious errors.