Accurately estimating human internal states, such as personality traits or behavioral patterns, is critical for enhancing the effectiveness of human–robot interaction, particularly in multi-agent settings. These insights are key in applications ranging from social navigation to autism diagnosis. However, prior methods are limited by scalability and passive observation, making real-time estimation in complex, multi-human settings difficult. In this work, we propose a practical method for active human personality estimation in crowds, with a focus on applications related to Autism Spectrum Disorder (ASD). Our method combines a personality-conditioned behavior model, based on the Eysenck 3-Factor theory, with an active robot information-gathering policy that triggers human behaviors through a receding-horizon planner. The robot’s belief about human personality is then updated via Bayesian inference. We demonstrate the effectiveness of our approach through proof-of-concept studies in simulation, user studies with typical adults, and preliminary experiments involving participants with ASD. Our results show that our method can scale to tens of humans and reduce personality estimation error by 29.2% and uncertainty by 79.9% in simulation compared to the passive baseline. User studies with typical adults confirm the method’s ability to generalize across complex personality distributions. Additionally, we explore its application in autism-related scenarios, demonstrating that the method can identify the difference between neurotypical and autistic behavior. The results suggest that our framework could serve as a foundation for future ASD-specific applications.

While yarn-level garments offer rich dynamic details and compelling visual realism compared to triangle-based models, their wide adoption is hindered by the immense computational cost due to the presence of a large number of degrees of freedom (DOFs). This paper proposes a novel simulation framework designed to enhance the performance and stability for numerical simulation of nonlinear, non-convex, and high-resolution knitwear. Our method generalizes the Douglas-Rachford Splitting (DRS) scheme to resolve the non-convex coupling between stretching, shearing, bending, twisting, and contacting at each yarn thread. A key contribution is a nested decomposition strategy that decouples the non-convex variational energy into independent and convex sub-problems. Such convexification improves solver robustness and removes the necessity for frequent line searches. We provide a theoretically grounded strategy for metric selection for each sub-problem, derived from an analysis of the convergence guarantee of DRS. Consequently, our method achieves close-to-optimal convergence along the nonlinear iterations rather than relying on ad-hoc parameter tuning. The paper also clarifies a formal connection between our generalized DRS and the ADMM (Alternating Direction Method of Multipliers) framework, extending the applicability of our analysis to a broader set of constrained dynamics problems. Experimental results demonstrate that our method robustly handles complex knitwear simulation scenes with superior efficiency, stability, and physical fidelity compared to existing methods. With a matrix-free GPU parallelization, our method allows an interactive simulation rate of knitwear of multi-million DOFs.

Robot simulators are indispensable tools across many fields, and recent research has significantly improved their functionality by incorporating additional gradient information. However, existing differentiable robot simulators suffer from non-differentiable singularities, when robots undergo substantial shape changes. To address this, we present the Shape-Differentiable Robot Simulator (SDRS), designed to be differentiable under significant robot shape changes. The core innovation of SDRS lies in its representation of robot shapes using a set of convex polyhedrons. This approach allows us to generalize smooth, penalty-based contact mechanics for interactions between any pair of convex polyhedrons. Using the separating hyperplane theorem, SDRS introduces a separating plane for each pair of contacting convex polyhedrons. This separating plane functions as a zero-mass auxiliary entity, with its state determined by the principle of least action. This setup ensures global differentiability, even as robot shapes undergo significant geometric and topological changes. To demonstrate the practical value of SDRS, we provide examples of robot co-design scenarios, where both robot shapes and control movements are optimized simultaneously.

Occlusion culling is a cornerstone of real-time rendering, particularly in mobile games where limited GPU bandwidth demands highly efficient scene management. At the heart of occlusion culling lies the use of simplified proxy geometry-called occluders-that approximate scene geometry for rapid visibility testing. However, producing high-quality occluders that are low in polygon count, conservative in coverage, and tightly aligned with the original geometry remains a manual and labor-intensive process. In this paper, we present a fast and fully automated two-stage approach for robust occluder generation tailored to real-world game assets. Our method begins with a novel strategy for inward offset mesh computation, followed by a conservative simplification step leveraging a new variant of Quadric Error Metrics (QEM). This approach effectively handles noisy and topologically complex inputs, generating production-ready occluders in seconds. Extensive experiments on a wide range of asset types demonstrate that our technique achieves aggressive triangle reduction while preserving critical occlusion fidelity. By offering a practical and scalable solution, our method bridges the gap between academic research and demanding needs for game development.

Mesh deformation is a fundamental tool in 3D content manipulation. Despite extensive prior research, existing approaches often suffer from low output quality, require significant manual tuning, or depend on data-intensive training. To address these limitations, we introduce a training-free, handle-based mesh deformation method. Our core idea is to leverage a Vision-Language Model (VLM) to interpret and manipulate a handle-based interface. We begin by applying cone singularity detection to identify a sparse set of potential handles. The VLM is then prompted to select both the deformable sub-parts of the mesh and the handles that best align with user instructions. Subsequently, we query the desired deformed positions of the selected handles in screen space. To reduce uncertainty inherent in VLM predictions, we aggregate the results from multiple camera views using a novel multi-view voting scheme. Across a suite of benchmarks, our method produces deformations that align more closely with user intent, as measured by CLIP and GPTEval3D scores, while introducing low distortion-quantified via membrane energy. In summary, our approach is highly automated and consistently delivers high-quality mesh deformations.

The motion planning problem involves finding a collision-free path from a robot's starting to its target configuration. Recently, self-supervised learning methods have emerged to tackle motion planning problems without requiring expensive expert demonstrations. They solve the Eikonal equation for training neural networks and lead to efficient solutions. However, these methods struggle in complex environments because they fail to maintain key properties of the Eikonal equation, such as optimal value functions and geodesic distances. To overcome these limitations, we propose a novel self-supervised temporal difference metric learning approach that solves the Eikonal equation more accurately and enhances performance in solving complex and unseen planning tasks. Our method enforces Bellman's principle of optimality over finite regions, using temporal difference learning to avoid spurious local minima while incorporating metric learning to preserve the Eikonal equation's essential geodesic properties. We demonstrate that our approach significantly outperforms existing self-supervised learning methods in handling complex environments and generalizing to unseen environments, with robot configurations ranging from 2 to 12 degrees of freedom (DOF).

Modular robotic structures simplify robot design and manufacturing by using standardized modules, enhancing flexibility and adaptability. However, the need for manual input in design and assembly limit their potential. Current methods to automate this process still require significant human effort and technical expertise. This paper introduces a novel approach that employs Large Language Models (LLMs) as intelligent agents to automate the creation of modular robotic structures. We decompose the modular robot creation task and develop two agents based on LLM to plan and assemble the modular robots from text prompts. By inputting a textual description, users can generate robot designs that are validated in both simulated and real-world environments. This method reduces the need for manual intervention and lowers the technical barrier to creating complex robotic systems.

We propose a Reinforcement Learning (RL) algorithm that combines several novel techniques to achieve more stable and robust control results for coupled solid-fluid systems. Our method utilizes the twin-delayed actor-critic algorithm to efficiently utilize off-policy data and achieve faster convergence. For more accurate estimations of the value function to guide the search of optimal policies, we use the Boltzmann softmax operator to reduce the bias of estimation. We further introduce a novel two-step Q-value estimator to reduce the well-known under-estimation issue. Finally, to mitigate the requirement of excessive exploration under sparse rewards, we propose the Fluid Effective Domain Guidance (FEDG) algorithm to guide policy exploration, where the policy for an easier task is trained jointly with that for a harder task. Put together, our framework achieves state-of-the-art performance in complex fluid-solid coupling control benchmarks, delivering stable and reliable performance in both 2D and 3D tasks over long horizons.

Approximate Convex Decomposition (ACD) aims to approximate complex 3D shapes with convex components, which is widely applied to create compact collision representations for real-time applications, including VR/AR, interactive games, and robotic simulations. Efficiency and optimality are critical for ACD algorithms in approximating large-scale, complex 3D shapes, enabling high-quality decompositions with minimal components. Unfortunately, existing methods either employ sub-optimal greedy strategies or rely on computationally intensive multi-step searches. In this work, we propose RL-ACD, a data-driven, reinforcement learning-based approach for efficient and near-optimal convex shape decomposition. We formulate ACD as a Markov Decision Process (MDP), where cutting planes are iteratively applied based on the current stage's mesh fragments rather than the entire fine-grained mesh, leading to a novel, efficient geometric encoding. To train near-optimal policies for ACD, we propose a novel dual-state Bellman loss and analyze its convergence using a Q-learning algorithm. Comprehensive evaluations across diverse datasets validate the efficiency and accuracy of RL-ACD for convex decomposition tasks. Our method outperforms the multi-step tree search by 15× in terms of computational speed, while reducing the number of resulting components by 16% compared to the current state-of-the-art greedy algorithms, significantly narrowing the sub-optimality gap and enhancing downstream task performance.

Hair cards remain a widely used representation for hair modeling in real-time applications, offering a practical trade-off between visual fidelity, memory usage, and performance. However, generating high-quality hair card models remains a challenging and labor-intensive task. This work presents an automated pipeline for converting strand-based hair models into hair card models with a limited number of cards and textures while preserving the hairstyle appearance. Our key idea is a novel differentiable representation where each strand is encoded as a projected 2D curve in the texture space, which enables end-to-end optimization with differentiable rendering while respecting the structures of the hair geometry. Based on this representation, we develop a novel algorithm pipeline, where we first cluster hair strands into initial hair cards and project the strands into the texture space. We then conduct a two-stage optimization, where our first stage optimizes the orientation of each hair card separately, and after strand projection, our second stage conducts joint optimization over the entire hair card model for fine-tuning. Our method is evaluated on a range of hairstyles, including straight, wavy, curly, and coily hair. To capture the appearance of short or coily hair, our method comes with support for hair caps and cross-card.

Predicting the state evolution of ultra high-dimensional, time-reversible fluid dynamic systems is a crucial but computationally expensive task. Existing physics-informed neural networks either incur high inference cost or cannot preserve the time-reversible nature of the underlying dynamics system. We propose a model-based approach to identify low-dimensional, time reversible, nonlinear fluid dynamic systems. Our method utilizes the symplectic structure of reduced Eulerian fluid and use stochastic Riemann optimization to obtain a low-dimensional bases that minimize the expected trajectory-wise dimension-reduction error over a given distribution of initial conditions. We show that such minimization is well-defined since the reduced trajectories are differentiable with respect to the subspace bases over the entire Grassmannian manifold, under proper choices of timestep sizes and numerical integrators. Finally, we propose a loss function measuring the trajectory-wise discrepancy between the original and reduced models. By tensor precomputation, we show that gradient information of such loss function can be evaluated efficiently over a long trajectory without time-integrating the high-dimensional dynamic system. Through evaluations on a row of simulation benchmarks, we show that our method reduces the discrepancy by 50-90 percent over conventional reduced models and we outperform PINNs by exactly preserving the time reversibility.

Strand-based hair simulations have recently become increasingly popular for a range of real-time applications. However, accurately simulating the full number of hair strands remains challenging. A commonly employed technique involves simulating a subset of guide hairs to capture the overall behavior of the hairstyle. Details are then enriched by interpolation using linear skinning. Hair interpolation enables fast real-time simulations but frequently leads to various artifacts during runtime. As the skinning weights are often pre-computed, substantial variations between the initial and deformed shapes of the hair can cause severe deviations in fine hair geometry. Straight hairs may become kinked, and curly hairs may become zigzags. This work introduces a novel physical-driven hair interpolation scheme that utilizes existing simulated guide hair data. Instead of directly operating on positions, we interpolate the internal forces from the guide hairs before efficiently reconstructing the rendered hairs based on their material model. We formulate our problem as a constraint satisfaction problem for which we present an efficient solution. Further practical considerations are addressed using regularization terms that regulate penetration avoidance and drift correction. We have tested various hairstyles to illustrate that our approach can generate visually plausible rendered hairs with only a few guide hairs and minimal computational overhead, amounting to only about 20% of conventional linear hair interpolation. This efficiency underscores the practical viability of our method for real-time applications.

Level of detail has been widely used in interactive computer graphics. In current industrial 3D modeling pipelines, artists rely on commercial software to generate highly detailed models with UV maps and then bake textures for low-poly counterparts. In these pipelines, each step is performed separately, leading to unsatisfactory visual appearances for low polygon count models. Moreover, existing texture baking techniques assume the low-poly mesh has a small geometric difference from the high-poly, which is often not true in practice, especially with extremely low poly count models. To alleviate the visual discrepancy of the low-poly mesh, we propose to jointly optimize UV mappings during texture baking, allowing for low-poly models to faithfully replicate the appearance of the high-poly even with large geometric differences. We formulate the optimization within a differentiable rendering framework, allowing the automatic adjustment of texture regions to encode appearance information. To compensate for view parallax when two meshes have large geometric differences, we introduce a spherical harmonic parallax mapping, which uses spherical harmonic functions to modulate per-texel UV coordinates based on the view direction. We evaluate the effectiveness and robustness of our approach on a dataset composed of online downloaded models, with varying complexities and geometric discrepancies. Our method achieves superior quality over state-of-the-art techniques and commercial solutions.

Simulating high-resolution cloth poses computational challenges in real-time applications. In the gaming industry, the proxy mesh technique offers an alternative, simulating a simplified low-resolution cloth geometry, proxy mesh. This proxy mesh's dynamics drive the detailed high-resolution geometry, visual mesh , through Linear Blended Skinning (LBS). However, generating a suitable proxy mesh with appropriate skinning weights from a given visual mesh is non-trivial, often requiring skilled artists several days for fine-tuning. This paper presents an automatic pipeline to convert an ill-conditioned highresolution visual mesh into a single-layer low-poly proxy mesh. Given that the input visual mesh may not be simulation-ready, our approach then simulates the proxy mesh based on specific use scenarios and optimizes the skinning weights, relying on differential skinning with several well-designed loss functions to ensure the skinned visual mesh appears plausible in the final simulation. We have tested our method on various challenging cloth models, demonstrating its robustness and effectiveness.

We present a semi-infinite program (SIP) solver for trajectory optimizations of general articulated robots. These problems are more challenging than standard nonlinear program by involving an infinite number of nonconvex, collision constraints. Prior SIP solvers based on constraint sampling cannot guarantee the satisfaction of all constraints. Instead, our method uses a conservative bound on articulated body motions to ensure the solution feasibility throughout the optimization procedure. We further use subdivision to adaptively reduce the error in conservative motion estimation. Combined, we prove that our SIP solver guarantees feasibility while approaching the optimal solution of SIP problems up to arbitrary user-provided precision. We demonstrate our method toward several trajectory optimization problems in simulation, including industrial robot arms and UAVs. The results demonstrate that our approach generates collision-free locally optimal trajectories within a couple of minutes.

Mesh repair is a long-standing challenge in computer graphics and related fields. Converting defective meshes into watertight manifold meshes can greatly benefit downstream applications such as geometric processing, simulation, fabrication, learning, and synthesis. In this work, by assuming the model is visually correct, we first introduce three visual measures for visibility, orientation, and openness, based on ray-tracing. We then present a novel mesh repair framework incorporating visual measures with several critical steps, i.e., open surface closing, face reorientation, and global optimization, to effectively repair meshes with defects (e.g., gaps, holes, self-intersections, degenerate elements, and inconsistent orientations) and preserve visual appearances. Our method reduces unnecessary mesh complexity without compromising geometric accuracy or visual quality while preserving input attributes such as UV coordinates for rendering. We evaluate our approach on hundreds of models randomly selected from ShapeNet and Thingi10K, demonstrating its effectiveness and robustness compared to existing approaches.

We present a lightweight, decentralized algorithm for navigating multiple nonholonomic agents through challenging environments with narrow passages. Our key idea is to allow agents to yield to each other in large open areas instead of narrow passages, to increase the success rate of conventional decentralized algorithms. At pre-processing time, our method computes a medial axis for the freespace. A reference trajectory is then computed and projected onto the medial axis for each agent. During run time, when an agent senses other agents moving in the opposite direction, our algorithm uses the medial axis to estimate a Point of Impact (POI) as well as the available area around the POI. If the area around the POI is not large enough for yielding behaviors to be successful, we shift the POI to nearby large areas by modulating the agent's reference trajectory and traveling speed. We evaluate our method on a row of 4 environments with up to 15 robots, and we find our method incurs a marginal computational overhead of 10–30 ms on average, achieving real-time performance. Afterward, our planned reference trajectories can be tracked using local navigation algorithms to achieve up to a 100% higher success rate over local navigation algorithms alone.

Deformable robots are notoriously difficult to model or control due to its high-dimensional configuration spaces. Direct trajectory optimization suffers from the curse-of-dimensionality and incurs a high computational cost, while learning-based controller optimization methods are sensitive to hyper-parameter tuning. To overcome these limitations, we hypothesize that high fidelity soft robots can be both simulated and controlled by restricting to low-dimensional spaces. Under such assumption, we propose a two-stage algorithm to identify such simulation- and control-spaces. Our method first identifies the so-called simulation-space that captures the salient deformation modes, to which the robot's governing equation is restricted. We then identify the control-space, to which control signals are restricted. We propose a multi-fidelity Riemannian Bayesian bilevel optimization to identify task-specific control spaces. We show that the dimension of control-space can be less than 10 for a high-DOF soft robot to accomplish walking and swimming tasks, allowing low-dimensional MPC controllers to be applied to soft robots with tractable computational complexity.

Optimizing the memory footprint of 3D models can have a major impact on the user experiences during real-time rendering and streaming visualization, where the major memory overhead lies in the high-resolution texture data. In this work, we propose a robust and automatic pipeline to content-aware, lossy compression for texture atlas. The design of our solution lies in two observations: 1) mapping multiple surface patches to the same texture region is seamlessly compatible with the standard rendering pipeline, requiring no decompression before any usage; 2) a texture image has background regions and salient structural features, which can be handled separately to achieve a high compression rate. Accordingly, our method contains joint operations of image segmentation, re-meshing, UV unwrapping, and texture baking. To evaluate the efficacy of our approach, we batch-processed a dataset containing 100 models collected online. On average, our method achieves a texture atlas compression ratio of 81.41% with an averaged PSNR and MS-SSIM scores of 40.90 and 0.98, a marginal error in visual appearance.

We propose a height-field-based real-time simulation method for sand and water mixtures. Inspired by the shallow-water assumption, our approach extends the governing equations to handle two-phase flows of sand and water using height fields. Our depth-integrated governing equations can model the elastoplastic behavior of sand, as well as sand-water-mixing phenomena such as friction, diffusion, saturation, and momentum exchange. We further propose an operator-splitting time integrator that is both GPU-friendly and stable under moderate time step sizes. We have evaluated our method on a set of benchmark scenarios involving large bodies of heterogeneous materials, where our GPU-based algorithm runs at real-time frame rates. Our method achieves a desirable trade-off between fidelity and performance, bringing an unprecedentedly immersive experience for real-time applications.

Topology Optimization (TO) is an essential tool for optimizing the structural robustness of load-bearing mechanical parts. An ideal TO solver should be computationally efficient for designers to preview the results, while ultimately converge to locally optimal designs. However, existing TO solvers either incur a high iterative cost or fail to provide the convergence guarantee. Borrowing ideas from recent advances in first-order bilevel optimization, we propose a new TO solver combining the Projected Gradient Descent (PGD) algorithm and inexact Finite Element Analysis (FEA). We further show that our method is convergent to a first-order critical point . Our proposed First-Order Bilevel Topology Optimization (FBTO) can solve several, important problems in the robot design paradigm, including TO under self-weight and multiple external loads. Finally, we evaluate and compare FBTO with prior TO solvers on a row of 2D and 3D problems.

We propose a robust re-meshing approach that can automatically generate visual-preserving low-poly meshes for any high-poly models found in the wild. Our method can be seamlessly integrated into current mesh-based 3D asset production pipelines. Given an input high-poly, our method proceeds in two stages: 1) Robustly extracting an offset surface mesh that is feature-preserving, and guaranteed to be watertight, manifold, and self-intersection free; 2) Progressively simplifying and flowing the offset mesh to bring it close to the input. The simplicity and the visual-preservation of the generated low-poly is controlled by a user-required target screen size of the input: decreasing the screen size reduces the element count of the low-poly but enlarges its visual difference from the input. We have evaluated our method on a subset of the Thingi10K dataset that contains models created by practitioners in different domains, with varying topological and geometric complexities. Compared to state-of-the-art approaches and widely used software, our method demonstrates its superiority in terms of the element count, visual preservation, geometry, and topology guarantees of the generated low-polys.

Lagrangian/Eulerian hybrid strand-based hair simulation techniques have quickly become a popular approach in VFX and real-time graphics applications. With Lagrangian hair dynamics, the inter-hair contacts are resolved in the Eulerian grid using the continuum method, i.e., the MPM scheme with the granular Drucker-Prager rheology, to avoid expensive collision detection and handling. This fuzzy collision handling makes the authoring process significantly easier. However, although current hair grooming tools provide a wide range of strand-based modeling tools for this simulation approach, the crucial sag-free initialization functionality remains often ignored. Thus, when the simulation starts, gravity would cause any artistic hairstyle to sag and deform into unintended and undesirable shapes. This paper proposes a novel four-stage sag-free initialization framework to solve stable quasistatic configurations for hybrid strand-based hair dynamic systems. These four stages are split into two global-local pairs. The first one ensures static equilibrium at every Eulerian grid node with additional inequality constraints to prevent stress from exiting the yielding surface. We then derive several associated closed-form solutions in the local stage to compute segment rest lengths, orientations, and particle deformation gradients in parallel. The second global-local step solves along each hair strand to ensure all the bend and twist constraints produce zero net torque on every hair segment, followed by a local step to adjust the rest Darboux vectors to a unit quaternion. We also introduce an essential modification for the Darboux vector to eliminate the ambiguity of the Cosserat rod rest pose in both initialization and simulation. We evaluate our method on a wide range of hairstyles, and our approach can only take a few seconds to minutes to get the rest quasistatic configurations for hundreds of hair strands. Our results show that our method successfully prevents sagging and has minimal impact on the hair motion during simulation.

Kinetic solvers for incompressible fluid simulation were designed to run efficiently on massively parallel architectures such as GPUs. While these lattice Boltzmann solvers have recently proven much faster and more accurate than the macroscopic Navier-Stokes-based solvers traditionally used in graphics, it systematically comes at the price of a very large memory requirement: a mesoscopic discretization of statistical mechanics requires over an order of magnitude more variables per grid node than most fluid solvers in graphics. In order to open up kinetic simulation to gaming and simulation software packages on commodity hardware, we propose a HighOrder Moment-Encoded Lattice-Boltzmann-Method solver which we coined HOME-LBM, requiring only the storage of a few moments per grid node, with little to no loss of accuracy in the typical simulation scenarios encountered in graphics. We show that our lightweight and lightspeed fluid solver requires three times less memory and runs ten times faster than state-of-the-art kinetic solvers, for a nearly-identical visual output.

We propose a learning-based controller for high-dimensional dynamic systems with coupled fluid and solid objects. The dynamic behaviors of such systems can vary across different simulators and the control tasks subject to changing requirements from users. Our controller features high versatility and can adapt to changing dynamic behaviors and multiple tasks without re-training, which is achieved by combining two training strategies. We use meta-reinforcement learning to inform the controller of changing simulation parameters. We further design a novel task representation, which allows the controller to adapt to continually changing tasks via hindsight experience replay. We highlight the robustness and generality of our controller on a row of dynamic-rich tasks, including scooping up solid balls from a water pool, in-air ball acrobatics using fluid spouts, and zero-shot transferring to unseen simulators and constitutive models. In all the scenarios, our controller consistently outperforms the plain multi-task reinforcement-learning baseline.

2D irregular shape packing is a necessary step to arrange UV patches of a 3D model within a texture atlas for memory-efficient appearance rendering in computer graphics. Being a joint, combinatorial decision-making problem involving all patch positions and orientations, this problem has well-known NP-hard complexity. Prior solutions either assume a heuristic packing order or modify the upstream mesh cut and UV mapping to simplify the problem, which either limits the packing ratio or incurs robustness or generality issues. Instead, we introduce a learning-assisted 2D irregular shape packing method that achieves a high packing quality with minimal requirements from the input. Our method iteratively selects and groups subsets of UV patches into near-rectangular super patches, essentially reducing the problem to bin-packing, based on which a joint optimization is employed to further improve the packing ratio. In order to efficiently deal with large problem instances with hundreds of patches, we train deep neural policies to predict nearly rectangular patch subsets and determine their relative poses, leading to linear time scaling with the number of patches. We demonstrate the effectiveness of our method on three datasets for UV packing, where our method achieves a higher packing ratio over several widely used baselines with competitive computational speed.

We study the problem of learning online packing skills for irregular 3D shapes , which is arguably the most challenging setting of bin packing problems. The goal is to consecutively move a sequence of 3D objects with arbitrary shapes into a designated container with only partial observations of the object sequence. We take physical realizability into account, involving physics dynamics and constraints of a placement. The packing policy should understand the 3D geometry of the object to be packed and make effective decisions to accommodate it in the container in a physically realizable way. We propose a Reinforcement Learning (RL) pipeline to learn the policy. The complex irregular geometry and imperfect object placement together lead to huge solution space. Direct training in such space is prohibitively data intensive. We instead propose a theoretically provable method for candidate action generation to reduce the action space of RL and the learning burden. A parameterized policy is then learned to select the best placement from the candidates. Equipped with an efficient method of asynchronous RL acceleration and a data preparation process of simulation-ready training sequences, a mature packing policy can be trained in a physics-based environment within 48 hours. Through extensive evaluation on a variety of real-life shape datasets and comparisons with state-of-the-art baselines, we demonstrate that our method outperforms the best-performing baseline on all datasets by at least 12.8% in terms of packing utility. We also release our datasets and source code to support further research in this direction. 1

Robot manipulation of multiple objects is an important topic for applications including warehouse automation, service robots performing cleaning, and large-scale object sorting. Although problems can range in complexity from a few objects to large disordered piles, autonomy remains a significant technical challenge due to the high-dimensional joint configuration space of the robot and all objects, the complex dynamics of object interaction, and the ambiguity and occlusion caused by clutter. This article surveys a broad range of classical and state-of-the-art research in multiobject manipulation and categorizes them along the dimensions of tasks, perception, predictive models, and decision-making algorithms. It also covers emerging trends and open problems faced in the ongoing effort to realize robust multiobject manipulation systems in practice.

We present a robust learning algorithm to detect and handle collisions in 3D deforming meshes. We first train a neural network to detect collisions and then use a numerical optimization algorithm to resolve penetrations guided by the network. Our learned collision handler can resolve collisions for unseen, high-dimensional meshes with thousands of vertices. To obtain stable network performance in such large and unseen spaces, we apply active learning by progressively inserting new collision data based on the network inferences. We automatically label these new data using an analytical collision detector and progressively fine-tune our detection networks. We evaluate our method for collision handling of complex, 3D meshes coming from several datasets with different shapes and topologies, including datasets corresponding to dressed and undressed human poses, cloth simulations, and human hand poses acquired using multi-view capture systems.

We present a centralized algorithm for labeled, disk-shaped Multi-Robot Path Planning (MPP) in a continuous planar workspace with polygonal boundaries. Our method automatically transform the continuous problem into a discrete, graph-based variant termed the pebble motion problem, which can be solved efficiently. To construct the underlying pebble graph, we identify inscribed circles in the workspace via a medial axis transform and organize robots into layers within each inscribed circle. We show that our layered pebble-graph enables collision-free motions, allowing all graph-restricted MPP instances to be feasible. MPP instances with continuous start and goal positions can then be solved via local navigations that route robots from and to graph vertices. We tested our method on several environments with high robot-packing densities (up to 61.6% of the workspace). For environments with narrow passages, such density violates the well-separated assumptions made by state-of-the-art MPP planners, while our method achieves an average success rate of 83%.

Abstract Occlusion culling has become a prevalent method in modern game engines. It can significantly reduce the rendering cost by using an approximate coarse mesh (occluder) for culling hidden objects. An ideal occluder should use as few faces as possible to represent the high‐resolution input mesh with a high culling accuracy. We address the open problem of automatic occluder generation for 3D building models with complex topology and interior structures. Our method first generates two coarse sets of faces via patch‐based and voxel‐based mesh simplification techniques. A metric‐guided selection algorithm chooses the best subset of faces to form the occluder, achieving a high occlusion rate and accuracy. Over an evaluation of 77 building models, our method compares favorably against state‐of‐the‐arts in terms of occlusion accuracy, occlusion rate, and face number.

We present LCollision, a learning-based method that synthesizes collision-free 3D human poses. At the crux of our approach is a novel deep architecture that simultaneously decodes new human poses from the latent space and predicts colliding body parts. These two components of our architecture are used as the objective function and surrogate hard constraints in a constrained optimization for collision-free human pose generation. A novel aspect of our approach is the use of a bilevel autoencoder that decomposes whole-body collisions into groups of collisions between localized body parts. By solving the constrained optimizations, we show that a significant amount of collision artifacts can be resolved. Furthermore, in a large test set of 2.5 × 10 6 randomized poses from SCAPE, our architecture achieves a collision-prediction accuracy of 94.1% with 80× speedup over exact collision detection algorithms. To the best of our knowledge, LCollision is the first approach that accelerates collision detection and resolves penetrations using a neural network.

Robot design is a time-consuming process involving repeated experiments in a variety of environments to optimize multiple, possibly conflicting performance metrics. Moreover, the optimal robot performance for a given design depends on how the robot adapts its behavior to its environment. We propose a multi-objective Bilevel Bayesian optimization (MO-BBO) technique to automate the process of form-behavior co-design. The approach expands the Pareto front of multiple metrics by simultaneously exploring the robot design and behavior. MO-BBO uses a bilevel optimization of the acquisition function with design and behavior parameters being the high- and low-level decision variables, respectively. In the low-level, we always choose environment-aware behaviors that maximize each metric. We evaluate MO-BBO in applications to grasping gripper design and bimanual arm placement, and show that our method can efficiently focus samples on the Pareto front and generate a diversity of designs.

UV radiation has been used as a disinfection strategy to deactivate a wide range of pathogens, but existing irradiation strategies do not ensure sufficient exposure of all environmental surfaces and/or require long disinfection times. We present a near-optimal coverage planner for mobile UV disinfection robots. The formulation optimizes the irradiation time efficiency, while ensuring that a sufficient dosage of radiation is received by each surface. The trajectory and dosage plan are optimized taking collision and light occlusion constraints into account. We propose a two-stage scheme to approximate the solution of the induced NP-hard optimization, and, for efficiency, perform key irradiance and occlusion calculations on a GPU. Empirical results show that our technique achieves more coverage for the same exposure time as strategies for existing UV robots, can be used to compare UV robot designs, and produces near-optimal plans.

Generating locally optimal UAV-trajectories is challenging due to the non-convex constraints of collision avoidance and actuation limits. We present the first local, optimization-based UAV-trajectory generator that simultane-ously guarantees validity and asymptotic optimality for known environments. Validity: Given a feasible initial guess, our algo-rithm guarantees the satisfaction of all constraints throughout the process of optimization. Asymptotic Optimality: We use an asymptotic exact piecewise approximation of the trajectory with an automatically adjustable resolution of its discretization. The trajectory converges under refinement to the first-order stationary point of the exact non-convex programming problem. Our method has additional practical advantages including joint optimality in terms of trajectory and time-allocation, and robustness to challenging environments as demonstrated in our experiments.

We present a planner for large-scale (un)labeled object sorting tasks, which uses two types of manipulation actions: overhead grasping and planar pushing. The grasping action offers completeness guarantee under mild assumptions, and the planar pushing is an acceleration strategy that moves multiple objects at once. We make two main contributions: (1) We propose a bilevel planning algorithm. Our high-level planner makes efficient, near-optimal choices between pushing and grasping actions based on a cost model. Our low-level planner computes one-step greedy pushing or grasping actions. (2) We propose a novel low-level push planner that can find one-step greedy pushing actions in a semi-discrete search space. The structure of the search space allows us to efficiently make decisions. We show that, for sorting up to 200 objects, our planner can find near-optimal actions within 10 seconds of computation on a desktop PC.

We present non-convex maximal dissipation principle (NMDP), a time integration scheme for articulated bodies with simultaneous contacts. Our scheme resolves contact forces via the maximal dissipation principle (MDP). Whereas prior MDP solvers assume linearized dynamics and integrate using the forward multistep scheme, we consider the coupled system of nonlinear Newton-Euler dynamics and MDP and integrate using the backward integration scheme. We show that the coupled system of equations can be solved efficiently using a novel projected gradient method with guaranteed convergence. We evaluate our method by predicting several locomotion trajectories for a quadruped robot. The results show that our NMDP scheme has several desirable properties including: (1) generalization to novel contact models; (2) stability under large timestep sizes; (3) consistent trajectory generation under varying timestep sizes.

We present a bilevel, contact-implicit trajectory optimization (TO) formulation that searches for robot trajectories with learned soft contact models. On the lower-level, contact forces are solved via a quadratic program (QP) with the maximum dissipation principle (MDP), based on which the dynamics constraints are formulated in the upper-level TO problem that uses direct transcription. Our method uses a contact model for granular media that is learned from physical experiments, but is general to any contact model that is stick-slip, convex, and smooth. We employ a primal interior-point method with a pre-specified duality gap to solve the lower-level problem, which provides robust gradient information to the upper-level problem. We evaluate our method by optimizing locomotion trajectories of a quadruped robot on various granular terrains offline, and show that we can obtain long-horizon walking gaits of high qualities.

We propose a decentralized, learning-based solution to the challenging problem of unlabeled multi-agent navigation among obstacles, where robots need to simultaneously tackle the problems of goal assignment, local collision avoidance, and navigation. Our method has each robot infer their desired action by communicating with each other as well as a set of position-fixed routers. The inference is carried out on a graph neural network (GNN) with both robot and router nodes. We train our GNN using imitation learning on a small group of robots, where we modify the centralized version of the concurrent goal assignment and planning algorithm (CAPT) as our expert. By sharing weights among all robots and routers, our model can scale to unseen environments with any number of possibly kinodynamic agents during test time. We have achieved a success rate of 91.2% and 85.6% for point and car-like robots, respectively. Source code will be publicly available upon the publication of the work.

We present a new method to generate optimal grasps for brittle and fragile objects using a novel stress- minimization (SM) metric. Our approach is designed for objects that are composed of homogeneous isotopic materials. Our SM metric measures the maximal resistible external wrenches that would not result in fractures in the target objects. In this paper, we propose methods to compute our new metric. We also use our SM metric to design optimal grasp planning algorithms. Finally, we compare the performance of our metric and conventional grasp metrics, including Q1,Q∞,QG11,QMSV,QVEW. Our experiments show that our SM metric takes into account the material characteristics and object shapes to indicate the fragile regions, where prior methods may not work well. We also show that the computational cost of our SM metric is on par with prior methods. Finally, we show that grasp planners guided by our metric can lower the probability of breaking target objects.

Given an articulated robot arm, we present a method to identify two regions with non-empty interiors. The first region is a subset of the configuration space where every point in the region is manipulable. The second region is a subset of the workspace where every point in the region is reachable by the end-effector. Our method expresses the kinematic state of the robot arm using the maximal coordinates, so that the kinematic constraints take polynomial forms. We then reformulate the optimization-based inverse kinematics (IK) algorithm as gradient flows. Finally, we use sum-of-squares (SOS) programming to certify the convergence of each gradient flow. Our main result shows that the feasibility of an SOS programming problem is a sufficient condition for the manipulability and reachability of the sublevel sets of polynomial functions. Our method can be used to certify manipulable or reachable regions by solving a set of linear matrix inequalities (LMIs) or to maximize the volume of a region by solving a set of bilinear matrix inequalities (BMIs). These identified regions can then be used in various motion planning problems as hard safety constraints.

We present an efficient algorithm for motion planning and controlling a robot system with a high number of degrees-of-freedom (DOF). These systems include high-DOF soft robots and articulated robots interacting with a deformable environment. We present a novel technique to accelerate the evaluations of the forward dynamics function by storing the results of costly computations in a hierarchical adaptive grid. Furthermore, we exploit the underactuated properties of the robot systems and build the grid in a low-dimensional space. Our approach approximates the forward dynamics function with guaranteed error bounds and can be used in optimization-based motion planning and reinforcement-learning-based feed-back control. We highlight the performance on two high-DOF robot systems: a line-actuated elastic robot arm and an underwater swimming robot in water. Compared to prior techniques based on exact dynamics evaluation, we observe one to two orders of magnitude improvement in the performance.

We present a learning-based method for representing grasp poses of a high-DOF hand using neural networks. Due to redundancy in such high-DOF grippers, there exists a large number of equally effective grasp poses for a given target object, making it difficult for the neural network to find consistent grasp poses. We resolve this ambiguity by generating an augmented dataset that covers many possible grasps for each target object and train our neural networks using a consistency loss function to identify a one-to-one mapping from objects to grasp poses. We further enhance the quality of neural-network-predicted grasp poses using a collision loss function to avoid penetrations. We use an object dataset that combines the BigBIRD Database, the KIT Database, the YCB Database, and the Grasp Dataset to show that our method can generate high-DOF grasp poses with higher accuracy than supervised learning baselines. The quality of the grasp poses is on par with the groundtruth poses in the dataset. In addition, our method is robust and can handle noisy object models such as those constructed from multi-view depth images, allowing our method to be implemented on a 25-DOF Shadow Hand hardware platform.

We present a method for planning the motion of arbitrarily-shaped volumetric deformable bodies or robots through complex environments. Such robots have very high-dimensional configuration spaces and we compute trajectories that satisfy the dynamics constraints using a two-stage learning method. First, we train a multitask controller parameterized using dynamic movement primitives (DMP), which encodes various locomotion or movement skills. Next, we train a neural-network controller to select the DMP task to navigate the robot through environments while avoiding obstacles. By combining the finite element method (FEM), model reduction, and contact invariant optimization (CIO), the DMP controller's parameters can be optimized efficiently using a gradient-based method, while the neural-network's parameters are optimized using Deep Q-Learning (DQL). This two-stage learning algorithm also allows us to reuse the trained DMP controller for different navigation tasks, such as moving through different environmental types and to different goal positions. Our results show that the learned motion planner can navigate swimming and walking deformable robots with thousands of DOFs at realtime.

We present a new time-integrator for modeling the frictional dynamics of articulated bodies. Our formulation represents the configuration of the articulated body using position variables and then uses those variables to model the friction forces between the articulated body and the environment. Our approach corresponds to a Newton-type optimization scheme that is guaranteed to converge so that it is stable with large timestep sizes. We evaluate the accuracy and stability of our time-integrator by comparing it with a conventional formulations based on the Newton-Euler equation and demonstrate the benefits on standard controller-optimization applications. We achieve 3-5 times speedup over a Newton-Euler-based simulator on a CPU. Our approach can be easily parallelized on a GPU and results in additional 4-15 times performance improvement.

We present a learning-based method to control a coupled 2D system involving both fluid and rigid bodies. Our approach is used to modify the fluid/rigid simulator's behavior by applying control forces only at the simulation domain boundaries. The rest of the domain, corresponding to the interior, is governed by the Navier-Stokes equation for fluids and Newton-Euler's equation for the rigid bodies. We represent our controller using a general neural-net, which is trained using deep reinforcement learning. Our formulation decomposes a control task into two stages: a precomputation training stage and an online generation stage. We utilize various fluid properties, e.g., the liquid's velocity field or the smoke's density field, to enhance the controller's performance. We set up our evaluation benchmark by letting controller drive fluid jets move on the domain boundary and allowing them to shoot fluids towards a rigid body to accomplish a set of challenging 2D tasks such as keeping a rigid body balanced, playing a two-player ping-pong game, and driving a rigid body to sequentially hit specified points on the wall. In practice, our approach can generate physically plausible animations.

We present an efficient spacetime optimization method to automatically generate animations for a general volumetric, elastically deformable body. Our approach can model the interactions between the body and the environment and automatically generate active animations. We model the frictional contact forces using contact invariant optimization and the fluid drag forces using a simplified model. To handle complex objects, we use a reduced deformable model and present a novel hybrid optimizer to search for the local minima efficiently. This allows us to use long-horizon motion planning to automatically generate animations such as walking, jumping, swimming, and rolling. We evaluate the approach on different shapes and animations, including deformable body navigation and combining with an open-loop controller for realtime forward simulation.

Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web. As an alternative, we propose a method that is able to compute a 3D model of a human body and its outfit from a single photograph with little human interaction. Our algorithm is not only able to capture the global shape and overall geometry of the clothing, it can also extract the physical properties (i.e., material parameters needed for simulation) of cloth. Unlike previous methods using full 3D information (i.e., depth, multi-view images, or sampled 3D geometry), our approach achieves garment recovery from a single-view image by using physical, statistical, and geometric priors and a combination of parameter estimation, semantic parsing, shape/pose recovery, and physics-based cloth simulation. We demonstrate the effectiveness of our algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications and for cloth animation on digital characters.

We present a novel motion planning algorithm for pouring a liquid body from a source to a target container. Our approach uses a receding-horizon optimization strategy that considers liquid dynamics and various other constraints. To handle liquid dynamics without costly fluid simulations, we use a neural network to infer a set of key liquid-related parameters from the observation of the current liquid configuration. To train the neural network, we generate a dataset of successful pouring examples using stochastic optimization in a problem-specific search space. These parameters are then used in the objective function for trajectory optimization. Our feedback motion planner achieves real-time performance, and we observe a high success rate in our simulated 2D and 3D liquid pouring benchmarks.

Abstract Hexahedral (hex‐) meshes are important for solving partial differential equations (PDEs) in applications of scientific computing and mechanical engineering. Many methods have been proposed aiming to generate hex‐meshes with high scaled Jacobians. While it is well established that a hex‐mesh should be inversion‐free (i.e. have a positive Jacobian measured at every corner of its hexahedron), it is not well‐studied that whether the scaled Jacobian is the most effective indicator of the quality of simulations performed on inversion‐free hex‐meshes given the existing dozens of quality metrics for hex‐meshes. Due to the challenge of precisely defining the relations among metrics, studying the correlations among different quality metrics and their correlations with the stability and accuracy of the simulations is a first and effective approach to address the above question. In this work, we propose a correlation analysis framework to systematically study these correlations. Specifically, given a large hex‐mesh dataset, we classify the existing quality metrics into groups based on their correlations, which characterizes their similarity in measuring the quality of hex‐elements. In addition, we rank the individual metrics based on their correlations with the accuracy and stability metrics for simulations that solve a number of elliptic PDE problems. Our preliminary experiments suggest that metrics that assess the conditioning of the elements are more correlated to the quality of solving elliptic PDEs than the others. Furthermore, an inversion‐free hex‐mesh with higher average quality (measured by any quality metrics) usually leads to a more accurate and stable computation of elliptic PDEs. To support our correlation study and address the lack of a publicly available large hex‐mesh dataset with sufficiently varying quality metric values, we also propose a two‐level perturbation strategy to generate the desired dataset from a small number of meshes to exclude the influences of element numbers, vertex connectivity, and volume sizes to our study.

We present a new method for editing smoke animations by directly deforming the grid used for simulation. We present a modification to the widely used semi-Lagrangian advection operator and use it to transfer the deformation from the grid to the smoke body. Our modified operator bends the smoke particle streamlines according to the deformation gradient. We demonstrate that the controlled smoke animation preserves the fine-grained vortical velocity components and incompressibility constraints, while conforming to the deformed grid. Moreover, our approach enables interactive 3D smoke animation editing by using a reduced-dimensional subspace. Overall, our method makes it possible to use current mesh editing tools to control the smoke body.

We present a novel algorithm to control the physically-based animation of smoke. Given a set of keyframe smoke shapes, we compute a dense sequence of control force fields that can drive the smoke shape to match several keyframes at certain time instances. Our approach formulates this control problem as a spacetime optimization constrained by partial differential equations. In order to compute the locally optimal control forces, we alternatively optimize the velocity fields and density fields using an alternating direction method of multiplier (ADMM) optimizer. In order to reduce the high complexity of multiple passes of fluid resimulation during velocity field optimization, we utilize the coherence between consecutive fluid simulation passes. We demonstrate the benefits of our approach by computing accurate solutions on 2D and 3D benchmarks. In practice, we observe up to an order of magnitude improvement over prior optimal control methods.

We present an optimization-based motion planning algorithm to compute a smooth, collision-free trajectory for a manipulator used to transfer a liquid from a source to a target container. We take into account fluid dynamics constraints as part of the trajectory computation. In order to avoid the high complexity of exact fluid simulation, we introduce a simplified dynamics model based on physically inspired approximations and system identification. Our optimization approach can incorporate various other constraints such as collision avoidance with obstacles, kinematic and dynamics constraints of the manipulator, and fluid dynamics characteristics. We demonstrate the performance of our planner on different benchmarks corresponding to various obstacles and container shapes. We also evaluate its accuracy by validating the motion plan using an accurate but computationally costly Navier-Stokes fluid simulation.

We present a stable and efficient simulator for deformable objects with collisions and contacts. For stability, an optimization derived from the implicit time integrator is solved in each timestep under the inequality constraints coming from collisions. To achieve fast convergence, we extend the MPRGP based solver from handling box constraints only to handling general linear constraints and prove its convergence. This generalization introduces a cost of solving dense linear systems in each step, but these systems can be reduced into diagonal ones for efficiency without affecting the general stability via pruning redundant collisions. Our solver is an order of magnitude faster, especially for elastic objects under large deformation compared with iterative constraint anticipation method (ICA), a typical method for stability. The efficiency, robustness and stability are further verified by our results.

In this paper, we propose a full featured and efficient subspace simulation method in the rotation-strain (RS) space for elastic objects. Sharply different from previous methods using the rotation-strain space, except for the ability to handle non-linear elastic materials and external forces, our method correctly formulates the kinetic energy, centrifugal and Coriolis forces which significantly reduces the dynamic artifacts. We show many techniques used in the Euclidean space methods, such as modal derivatives, polynomial and cubature approximation, can be adapted to our RS simulator. Carefully designed experiments show that the equation of motion in RS space has less non-linearity than its Euclidean counterpart, and as a consequence, our method has great advantages of lower dimension and computational complexity than state-of-the-art methods in the Euclidean space.

Animation techniques for controlling liquid simulation are challenging: they commonly require carefully setting initial and boundary conditions or performing a costly numerical optimization scheme against user-provided keyframes or animation sequences. Either way, the whole process is laborious and computationally expensive. We introduce a novel method to provide intuitive and interactive control of liquid simulation. Our method enables a user to locally edit selected keyframes and automatically propagates the editing in a nearby temporal region using geometric deformation. We formulate our local editing techniques as a small-scale nonlinear optimization problem which can be solved interactively. With this uniformed formulation, we propose three editing metaphors, including (i) sketching local fluid features using a few user strokes, (ii) dragging a local fluid region, and (iii) controlling a local shape with a small mesh patch. Finally, we use the edited liquid animation to guide an offline high-resolution simulation to recover more surface details. We demonstrate the intuitiveness and efficacy of our method in various practical scenarios.

Image-space line integral convolution (LIC) is a popular scheme for visualizing surface vector fields due to its simplicity and high efficiency. To avoid inconsistencies or color blur during the user interactions, existing approaches employ surface parameterization or 3D volume texture schemes. However, they often require expensive computation or memory cost, and cannot achieve consistent results in terms of both the granularity and color distribution on different scales. This paper introduces a novel image-space surface flow visualization approach that preserves the coherence during user interactions. To make the noise texture under different viewpoints coherent, we propose to precompute a sequence of mipmap noise textures in a coarse-to-fine manner for consistent transition, and map the textures onto each triangle with randomly assigned and constant texture coordinates. Further, a standard image-space LIC is performed to generate the flow texture. The proposed approach is simple and GPU-friendly, and can be easily combined with various texture-based flow visualization techniques. By leveraging viewpoint-dependent backward tracing and mipmap noise phase, our method can be incorporated with the image-based flow visualization (IBFV) technique for coherent visualization of unsteady flows. We demonstrate consistent and highly efficient flow visualization on a variety of data sets.

Abstract In fluid animation, wake is one of the most important phenomena usually seen when an object is moving relative to the flow. However, in current shallow water simulation for interactive applications, this effect is greatly smeared out. In this paper, we present a method to efficiently synthesize these wakes. We adopt a generalized SPH method for shallow water simulation and two way solid fluid coupling. In addition, a 2D discrete vortex method is used to capture the detailed wake motions behind an obstacle, enriching the motion of SWE simulation. Our method is highly efficient since only 2D simulation is required. Moreover, by using a physically inspired procedural approach for particle seeding, DVM particles are only created in the wake region. Therefore, very few particles are required while still generating realistic wake patterns. When coupled with SWE, we show that these patterns can be seen using our method with marginal overhead.