Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.

Teleoperated robotic characters can perform expressive interactions with humans, relying on the operators’ experience and social intuition. In this work, we propose to create autonomous interactive robots, by training a model to imitate operator data. Our model is trained on a dataset of human-robot interactions, where an expert operator is asked to vary the interactions and mood of the robot, while the operator commands as well as the pose of the human and robot are recorded. Our approach learns to predict continuous operator commands through a diffusion process and discrete commands through a classifier, all unified within a single transformer architecture. We evaluate the resulting model in simulation and with a user study on the real system. We show that our method enables simple autonomous human-robot interactions that are comparable to the expert-operator baseline, and that users can recognize the different robot moods as generated by our model. Finally, we demonstrate a zero-shot transfer of our model onto a different robotic platform with the same operator interface.

We present a constrained Rigid Body Dynamics (RBD) that guarantees satisfaction of kinematic constraints, enabling direct simulation of complex mechanical systems with arbitrary kinematic structures. To ensure constraint satisfaction, we use an implicit integration scheme. For this purpose, we derive compatible dynamic equations expressed through the quaternion time derivative, adopting an additive approach to quaternion updates instead of a multiplicative one, while enforcing quaternion unit-length as a constraint. We support all joints between rigid bodies that restrict subsets of the three translational or three rotational degrees of freedom, including position- and force-based actuation. Their constraints are formulated such that Lagrange multipliers are interpretable as joint forces and torques. We discuss a unified solution strategy for systems with redundant constraints, overactuation, and passive degrees of freedom, by eliminating redundant constraints and navigating the subspaces spanned by multipliers. As our method uses a standard additive update, we can interface with unconditionally-stable implicit integrators. Moreover, the simulation can readily be made differentiable as we show with examples.

Abstract Recent progress in physics‐based character control has made it possible to learn policies from unstructured motion data. However, it remains challenging to train a single control policy that works with diverse and unseen motions, and can be deployed to real‐world physical robots. In this paper, we propose a two‐stage technique that enables the control of a character with a full‐body kinematic motion reference, with a focus on imitation accuracy. In a first stage, we extract a latent space encoding by training a variational autoencoder, taking short windows of motion from unstructured data as input. We then use the embedding from the time‐varying latent code to train a conditional policy in a second stage, providing a mapping from kinematic input to dynamics‐aware output. By keeping the two stages separate, we benefit from self‐supervised methods to get better latent codes and explicit imitation rewards to avoid mode collapse. We demonstrate the efficiency and robustness of our method in simulation, with unseen user‐specified motions, and on a bipedal robot, where we bring dynamic motions to the real world.

Procedural animation has seen widespread use in the design of expressive walking gaits for virtual characters. While similar tools could breathe life into robotic characters, existing techniques are largely unaware of the kinematic and dynamic constraints imposed by physical robots. In this paper, we propose a system for the artist-directed authoring of stylized bipedal walking gaits, tailored for execution on robotic characters. The artist interfaces with an interactive editing tool that generates the desired character motion in realtime, either on the physical or simulated robot, using a model-based control stack. Each walking style is encoded as a set of sample parameters which are translated into whole-body reference trajectories using the proposed procedural animation technique. In order to generalize the stylized gait over a continuous range of input velocities, we employ a phase-space blending strategy that interpolates a set of example walk cycles authored by the animator while preserving contact constraints. To demonstrate the utility of our approach, we animate gaits for a custom, free-walking robotic character, and show, with two additional in-simulation examples, how our procedural animation technique generalizes to bipeds with different degrees of freedom, proportions, and mass distributions.

Legged robots are designed to perform highly dynamic motions. However, it remains challenging for users to retarget expressive motions onto these complex systems. In this paper, we present a Differentiable Optimal Control (DOC) framework that facilitates the transfer of rich motions from either animals or animations onto these robots. Interfacing with either motion capture or animation data, we formulate retargeting objectives whose parameters make them agnostic to differences in proportions and numbers of degrees of freedom between input and robot. Optimizing these parameters over the manifold spanned by optimal state and control trajectories, we minimize the retargeting error. We demonstrate the utility and efficacy of our modeling by applying DOC to a Model-Predictive Control (MPC) formulation, showing retargeting results for a family of robots of varying proportions and mass distribution. With a hardware deployment, we further show that the retargeted motions are physically feasible, while MPC ensures that the robots retain their capability to react to unexpected disturbances.

The kinematic motion of a robotic character is defined by its mechanical joints and actuators that restrict the relative motion of its rigid components. Designing robots that perform a given target motion as closely as possible with a fixed number of actuated degrees of freedom is challenging, especially for robots that form kinematic loops. In this paper, we propose a technique that simultaneously solves for optimal design and control parameters for a robotic character whose design is parameterized with configurable joints. At the technical core of our technique is an efficient solution strategy that uses dynamic programming to solve for optimal state, control, and design parameters, together with a strategy to remove redundant constraints that commonly exist in general robot assemblies with kinematic loops. We demonstrate the efficacy of our approach by either editing the design of an existing robotic character, or by optimizing the design of a new character to perform a desired motion.

This paper presents an interactive design system that allows the user to create and fabricate stylized sculptures in water-based clay, using a standard 6-axis robot arm. This system facilitates the materialization of abstract design intentions into clay, through the algorithmic formulation of sculpting styles, the optimal path planning of the sculpting toolpaths, and a subtractive robotic fabrication process using customized tools. Unlike other precision-driven fabrication technologies, the authors embrace artistic uncertainty by conducting manual and robotic sculpting experiments and incorporating prominent parameters that affect the fabrication quality. The versatility of the described approach is demonstrated by designing a series of sculpting styles over a wide range of 3D models and robotically fabricating them in clay. Additionally, the paper explores various strategies for designing stylized robotic sculpting patterns by generating toolpaths informed by different techniques.

This article introduces a computational approach for routing thin artificial muscle actuators through hyperelastic soft robots, in order to achieve a desired deformation behavior. Provided with a robot design and a set of example deformations, we continuously co-optimize the routing of actuators, and their actuation, to approximate example deformations as closely as possible. We introduce a data-driven model for McKibben muscles, modeling their contraction behavior when embedded in a silicone elastomer matrix. To enable the automated routing, a differentiable hyperelastic material simulation is presented. Because standard finite elements are not differentiable at element boundaries, we implement a moving least squares formulation, making the deformation gradient twice differentiable. Our robots are fabricated in a two-step molding process, with the complex mold design steps automated. While most soft robotic designs utilize bending, we study the use of our technique in approximating twisting deformations on a bar example. To demonstrate the efficacy of our technique in soft robotic design, we show a continuum robot, a tentacle, and a four-legged walking robot.

Abstract We present a computational technique that aids with the design of structurally‐sound metal frames, tailored for robotic fabrication using an existing process that integrate automated bar bending, welding, and cutting. Aligning frames with structurally‐favorable orientations, and decomposing models into fabricable units, we make the fabrication process scale‐invariant, and frames globally align in an aesthetically‐pleasing and structurally‐informed manner. Relying on standard analysis of frames, we then co‐optimize the shape and topology of bars at the local unit level. At this level, we minimize combinations of functional and aesthetic objectives under strict fabrication constraints that model the assembly of discrete sets of bent bars. We demonstrate the capabilities of our global‐to‐local approach on four robotically‐constructed examples.

We present a differentiable dynamics solver that is able to handle frictional contact for rigid and deformable objects within a unified framework. Through a principled mollification of normal and tangential contact forces, our method circumvents the main difficulties inherent to the non-smooth nature of frictional contact. We combine this new contact model with fully-implicit time integration to obtain a robust and efficient dynamics solver that is analytically differentiable. In conjunction with adjoint sensitivity analysis, our formulation enables gradient-based optimization with adaptive trade-offs between simulation accuracy and smoothness of objective function landscapes. We thoroughly analyse our approach on a set of simulation examples involving rigid bodies, visco-elastic materials, and coupled multi-body systems. We furthermore showcase applications of our differentiable simulator to parameter estimation for deformable objects, motion planning for robotic manipulation, trajectory optimization for compliant walking robots, as well as efficient self-supervised learning of control policies.

We present a system for fast and robust handovers with a robot character, together with a user study investigating the effect of robot speed and reaction time on perceived interaction quality. The system can match and exceed human speeds and confirms that users prefer human-level timing. The system has the appearance of a robot character, with a bear-like head and a soft anthropomorphic hand and uses Bézier curves to achieve smooth minimum-jerk motions. Fast timing is enabled by low latency motion capture and real-time trajectory generation: the robot initially moves towards an expected handover location and the trajectory is updated on-the-fly to converge smoothly to the actual handover location. A hybrid automaton provides robustness to failure and unexpected human actions. In a 3x3 user study, we vary the speed of the robot and add variable sensorimotor delays. We evaluate the social perception of the robot using the Robot Social Attribute Scale (RoSAS). Inclusion of a small delay, mimicking the delay of the human sensorimotor system, leads to an improvement in perceived qualities over both no delay and long delay conditions. Specifically, with no delay the robot is perceived as more discomforting, and with a long delay it is perceived as less warm.

We propose a novel generic shape optimization method for CAD models based on the eXtended Finite Element Method (XFEM). Our method works directly on the intersection between the model and a regular simulation grid, without the need to mesh or remesh, thus removing a bottleneck of classical shape optimization strategies. This is made possible by a novel hierarchical integration scheme that accurately integrates finite element quantities with sub-element precision. For optimization, we efficiently compute analytical shape derivatives of the entire framework, from model intersection to integration rule generation and XFEM simulation. Moreover, we describe a differentiable projection of shape parameters onto a constraint manifold spanned by user-specified shape preservation, consistency, and manufacturability constraints. We demonstrate the utility of our approach by optimizing mass distribution, strength-to-weight ratio, and inverse elastic shape design objectives directly on parameterized 3D CAD models.

Creating animations for robotic characters is very challenging due to the constraints imposed by their physical nature. In particular, the combination of fast motions and unavoidable structural deformations leads to mechanical oscillations that negatively affect their performances. Our goal is to automatically transfer motions created using traditional animation software to robotic characters while avoiding such artifacts. To this end, we develop an optimization-based, dynamics-aware motion retargeting system that adjusts an input motion such that visually salient low-frequency, large amplitude vibrations are suppressed. The technical core of our animation system consists of a differentiable dynamics simulator that provides constraint-based two-way coupling between rigid and flexible components. We demonstrate the efficacy of our method through experiments performed on a total of five robotic characters including a child-sized animatronic figure that features highly dynamic drumming and boxing motions.

Abstract Facial performance capture is the primary method for generating facial animation in video games, feature films and virtual environments, and recent advances have produced very compelling results. Still, one of the most challenging regions is the mouth, which often contains systematic errors due to the complex appearance and occlusion/dis‐occlusion of the lips. We present a novel user‐guided approach to correcting these common lip shape errors present in traditional capture systems. Our approach is to allow a user to manually correct a small number of problematic frames, and then our system learns the types of corrections desired and automatically corrects the entire performance. As correcting even a single frame using traditional 3D sculpting tools can be time consuming and require great skill, we also propose a simple and fast 2D sketch‐based method for generating plausible lip corrections for the problematic key frames. We demonstrate our results on captured performances of three different subjects, and validate our method with an additional sequence that contains ground truth lip reconstructions.

Large-scale binder jetting provides a promising alternative to manual sculpting of sandstone. The weak build material, however, severely limits its use in architectural ornamentation. We propose a structural optimization that jointly optimizes an ornament's strength-to-weight ratio and balance under self-weight, thermal, wind, and live loads. To account for the difference in the tensile and compressive strength of the build material, we turn the Bresler-Pister criterion into a failure potential, measuring the distance to failure. Integrated into an XFEM-based level set formulation, we minimize this potential by changing the topology and shape of the internal structure. To deal with uncertainties in the location of live loads, and the direction of wind loads, we first estimate loads that lead to the weakest structure, then minimize the potential of failure under identified worst-case loads. With the help of first-order optimality constraints, we unify our worst-case load estimation and structural optimization into a continuous optimization. We demonstrate applications in art, furniture design, and architectural ornamentation with three large-scale 3D printed examples.

Elastically deforming wire structures are lightweight, durable, and can be bent within minutes using CNC bending machines. We present a computational technique for the design of kinetic wire characters, tailored for fabrication on consumer-grade hardware. Our technique takes as input a network of curves or a skeletal animation, then estimates a cable-driven, compliant wire structure which matches user-selected targets or keyframes as closely as possible. To enable large localized deformations, we shape wire into functional spring-like entities at a discrete set of locations. We first detect regions where changes to local stiffness properties are needed, then insert bendable entities of varying shape and size. To avoid a discrete optimization, we first optimize stiffness properties of generic, non-fabricable entities which capture well the behavior of our bendable designs. To co-optimize stiffness properties and cable forces, we formulate an equilibrium-constrained minimization problem, safeguarding against inelastic deformations. We demonstrate our method on six fabricated examples, showcasing rich behavior including large deformations and complex, spatial motion.

Handshakes are common greetings, and humans therefore have strong priors of what a handshake should feel like. This makes it challenging to create compelling and realistic human-robot handshakes, necessitating the consideration of human haptic perception in the design of robot hands. At its most basic level, haptic perception is encoded by contact points and contact pressure distributions on the skin. This motivates our work on measuring the contact area and contact pressure in human handshaking interactions. We present two benchmarking experiments in this regard, measuring the contact locations in human-human/human-robot handshaking and the contact pressure distribution for handshakes with a sensorized palm. We present results from human studies with the benchmarking experiments, providing a baseline for comparison with robot hands as well as presenting new insights into human handshaking. We also show initial work in using these results for the evaluation of robot hands, and progressing towards iterative design of robot hands optimized for social hand interactions.

Oftentimes facial animation is created separately from overall body motion. Since convincing facial animation is challenging enough in itself, artists tend to create and edit the face motion in isolation. Or if the face animation is derived from motion capture, this is typically performed in a mo-cap booth while sitting relatively still. In either case, recombining the isolated face animation with body and head motion is non-trivial and often results in an uncanny result if the body dynamics are not properly reflected on the face (e.g. the bouncing of facial tissue when running). We tackle this problem by introducing a simple and intuitive system that allows to add physics to facial blendshape animation. Unlike previous methods that try to add physics to face rigs, our method preserves the original facial animation as closely as possible. To this end, we present a novel simulation framework that uses the original animation as per-frame rest-poses without adding spurious forces. As a result, in the absence of any external forces or rigid head motion, the facial performance will exactly match the artist-created blendshape animation. In addition we propose the concept of blendmaterials to give artists an intuitive means to account for changing material properties due to muscle activation. This system allows to automatically combine facial animation and head motion such that they are consistent, while preserving the original animation as closely as possible. The system is easy to use and readily integrates with existing animation pipelines.

We present a computational tool for designing compliant mechanisms. Our method takes as input a conventional, rigidly-articulated mechanism defining the topology of the compliant design. This input can be both planar or spatial, and we support a number of common joint types which, whenever possible, are automatically replaced with parameterized flexures. As the technical core of our approach, we describe a number of objectives that shape the design space in a meaningful way, including trajectory matching, collision avoidance, lateral stability, resilience to failure, and minimizing motor torque. Optimal designs in this space are obtained as solutions to an equilibrium-constrained minimization problem that we solve using a variant of sensitivity analysis. We demonstrate our method on a set of examples that range from simple four-bar linkages to full-fledged animatronics, and verify the feasibility of our designs by manufacturing physical prototypes.

We present a method for designing and fabricating MetaSilicones ---composite silicone rubbers that exhibit desired macroscopic mechanical properties. The underlying principle of our approach is to inject spherical inclusions of a liquid dopant material into a silicone matrix material. By varying the number, size, and locations of these inclusions as well as their material, a broad range of mechanical properties can be achieved. The technical core of our approach is formed by an optimization algorithm that, combining a simulation model based on extended finite elements (XFEM) and sensitivity analysis, computes inclusion distributions that lead to desired stiffness properties on the macroscopic level. We explore the design space of MetaSilicone on an extensive set of simulation experiments involving materials with optimized uni- and bi-directional stiffness, spatially-graded properties, as well as multi-material composites. We present validation through standard measurements on physical prototypes, which we fabricate on a modified filament-based 3D printer, thus combining the advantages of digital fabrication with the mechanical performance of silicone elastomers.

We present a novel optimization-based algorithm for the design and fabrication of customized, deformable input devices, capable of continuously sensing their deformation. We propose to embed piezoresistive sensing elements into flexible 3D printed objects. These sensing elements are then utilized to recover rich and natural user interactions at runtime. Designing such objects is a challenging and hard problem if attempted manually for all but the simplest geometries and deformations. Our method simultaneously optimizes the internal routing of the sensing elements and computes a mapping from low-level sensor readings to user-specified outputs in order to minimize reconstruction error. We demonstrate the power and flexibility of the approach by designing and fabricating a set of flexible input devices. Our results indicate that the optimization-based design greatly outperforms manual routings in terms of reconstruction accuracy and thus interaction fidelity.

We present a method for interactive editing of planar linkages. Given a working linkage as input, the user can make targeted edits to the shape or motion of selected parts while preserving other, e.g., functionally-important aspects. In order to make this process intuitive and efficient, we provide a number of editing tools at different levels of abstraction. For instance, the user can directly change the structure of a linkage by displacing joints, edit the motion of selected points on the linkage, or impose limits on the size of its enclosure. Our method safeguards against degenerate configurations during these edits, thus ensuring the correct functioning of the mechanism at all times. Linkage editing poses strict requirements on performance that standard approaches fail to provide. In order to enable interactive and robust editing, we build on a symbolic kinematics approach that uses closed-form expressions instead of numerical methods to compute the motion of a linkage and its derivatives. We demonstrate our system on a diverse set of examples, illustrating the potential to adapt and personalize the structure and motion of existing linkages. To validate the feasibility of our edited designs, we fabricated two physical prototypes.

Spinning tops and yo-yos have long fascinated cultures around the world with their unexpected, graceful motions that seemingly elude gravity. We present an algorithm to generate designs for spinning objects by optimizing rotational dynamics properties. As input, the user provides a solid 3D model and a desired axis of rotation. Our approach then modifies the mass distribution such that the principal directions of the moment of inertia align with the target rotation frame. We augment the model by creating voids inside its volume, with interior fill represented by an adaptive multi-resolution voxelization. The discrete voxel fill values are optimized using a continuous, nonlinear formulation. Further, we optimize for rotational stability by maximizing the dominant principal moment. We extend our technique to incorporate deformation and multiple materials for cases where internal voids alone are insufficient. Our method is well-suited for a variety of 3D printed models, ranging from characters to abstract shapes. We demonstrate tops and yo-yos that spin surprisingly stably despite their asymmetric appearance.

Articulated deformable characters are widespread in computer animation. Unfortunately, we lack methods for their automatic fabrication using modern additive manufacturing (AM) technologies. We propose a method that takes a skinned mesh as input, then estimates a fabricatable single-material model that approximates the 3D kinematics of the corresponding virtual articulated character in a piecewise linear manner. We first extract a set of potential joint locations. From this set, together with optional, user-specified range constraints, we then estimate mechanical friction joints that satisfy inter-joint non-penetration and other fabrication constraints. To avoid brittle joint designs, we place joint centers on an approximate medial axis representation of the input geometry, and maximize each joint's minimal cross-sectional area. We provide several demonstrations, manufactured as single, assembled pieces using 3D printers.

This paper introduces a data-driven process for designing and fabricating materials with desired deformation behavior. Our process starts with measuring deformation properties of base materials. For each base material we acquire a set of example deformations, and we represent the material as a non-linear stress-strain relationship in a finite-element model. We have validated our material measurement process by comparing simulations of arbitrary stacks of base materials with measured deformations of fabricated material stacks. After material measurement, our process continues with designing stacked layers of base materials. We introduce an optimization process that finds the best combination of stacked layers that meets a user's criteria specified by example deformations. Our algorithm employs a number of strategies to prune poor solutions from the combinatorial search space. We demonstrate the complete process by designing and fabricating objects with complex heterogeneous materials using modern multi-material 3D printers.

This paper introduces a data-driven representation and modeling technique for simulating non-linear heterogeneous soft tissue. It simplifies the construction of convincing deformable models by avoiding complex selection and tuning of physical material parameters, yet retaining the richness of non-linear heterogeneous behavior. We acquire a set of example deformations of a real object, and represent each of them as a spatially varying stress-strain relationship in a finite-element model. We then model the material by non-linear interpolation of these stress-strain relationships in strain-space. Our method relies on a simple-to-build capture system and an efficient run-time simulation algorithm based on incremental loading, making it suitable for interactive computer graphics applications. We present the results of our approach for several non-linear materials and biological soft tissue, with accurate agreement of our model to the measured data.

The method of Moving Least Squares (MLS) is a popular framework for reconstructing continuous functions from scattered data due to its rich mathematical properties and well-understood theoretical foundations. This paper applies MLS to volume rendering, providing a unified mathematical framework for ray casting of scalar data stored over regular as well as irregular grids. We use the MLS reconstruction to render smooth isosurfaces and to compute accurate derivatives for high-quality shading effects. We also present a novel, adaptive preintegration scheme to improve the efficiency of the ray casting algorithm by reducing the overall number of function evaluations, and an efficient implementation of our framework exploiting modern graphics hardware. The resulting system enables high-quality volume integration and shaded isosurface rendering for regular and irregular volume data.