Scene-aware Complementary Item Retrieval (CIR) is a challenging task which requires to generate a set of compatible items across domains. Due to the subjectivity, it is difficult to set up a rigorous standard for both data collection and learning objectives. To address this challenging task, we propose a visual compatibility concept, composed of similarity (resembling in color, geometry, texture, and etc.) and complementarity (different items like table vs chair completing a group). Based on this notion, we propose a compatibility learning framework, a category-aware Flexible Bidirectional Transformer (FBT), for visual ``scene-based set compatibility reasoning'' with the cross-domain visual similarity input and auto-regressive complementary item generation. We introduce a ``Flexible Bidirectional Transformer (FBT),'' consisting of an encoder with flexible masking, a category prediction arm, and an auto-regressive visual embedding prediction arm. And the inputs for FBT are cross-domain visual similarity invariant embeddings, making this framework quite generalizable. Furthermore, our proposed FBT model learns the inter-object compatibility from a large set of scene images in a self-supervised way. Compared with the SOTA methods, this approach achieves up to 5.3% and 9.6% in FITB score and 22.3% and 31.8% SFID improvement on fashion and furniture, respectively.

Over the last decade, a plethora of soft robotic devices have been proposed for the execution of complex grasping and dexterous manipulation tasks. Tasks requiring such increased dexterity are typically executed using fully-actuated, rigid end-effectors equipped with sophisticated sensing and controlled with complex control laws. The new class of soft robotic devices offers an alternative to the traditional end-effectors and facilitates the development of robotic grasping and manipulation solutions that are lightweight, safe to interact with, affordable, and easy to use and control. Within the class of soft robotic grippers and hands, promising recent developments were made in ultra-affordable, even disposable mechanisms based on origami and kirigami structures. This paper proposes a new kirigami-inspired robotic gripper geometry employing compression-based actuation. The compression actuation fundamentally differentiates this new design class from previous kirigami grippers, resulting in more compact robotic grippers with superior grasping capabilities. In particular, we investigate how the shapes of the internal cuts of the kirigami geometries can affect the gripper performance in terms of force exertion and grasping capabilities. A series of experiments are conducted to understand better the working principles behind this new type of kirigami grippers and experimentally validate their efficacy in the execution of complex, everyday life tasks. Further demonstrations of the gripper's capabilities include the pick-and-placing of human hair, egg yolk, and even liquids.

Soft robotic devices have been popular in handling intricate grasping and dexterous manipulation tasks, serving as an alternative to conventional, rigid end-effectors. These devices are relatively simple, lightweight, and cost-effective. Recently, kirigami based structures have been used to create low-cost and disposable soft robotic grippers and hands. These grippers undergo a complex post-contact reconfiguration and conform to an object's shape and size during grasping. In this paper, we explore this new class of soft robotic grippers by utilising them for single-grasp object classification and grasping force estimation. We install simplistic sensors on both the gripper and the actuation system to estimate the state of the kirigami gripper, and the collected data features are employed to train Random Forest models for identifying the grasped object. The classifier trained exhibits a high accuracy of 98 % in discriminating objects of various shapes. When handling food items, the classifier achieves an accuracy of 94 %, while in classifying transparent objects, the classifier obtained again a high accuracy of 97 %. Finally, object-specific force estimation models are triggered based on the classification decision of the Random Forest model to estimate the grasping force exerted by the gripper. These positive outcomes demonstrate the kirigami based robotic gripper's potential for object classification in a variety of circumstances, particularly where vision systems are not available or not reliable.

Unmanned aerial vehicles (UAVs) have revolutionised various industries, such as agriculture, remote sensing, and infrastructure inspection. To explore new designs and improve UAV flight performance, roboticists are seeking inspiration from nature. In this paper, we present a bioinspired tailsitter UAV utilizing shape-morphing wings with aerofoil-shaped artificial feathers. The design of the UAV is inspired by the shape and motion of bird wings, which can change their shape and span to adapt to different flight conditions. The pigeon's wing skeletal structure serves as the basis for the design, and the wing was developed to be fully tendon-driven employing a single motor for each side. The wings can contract and extend, resulting in a contraction ratio of 49% of the extended wing span. In hovering flight mode, the wing contraction shows a 42% decrease in drag for improved wind disturbance rejection. Wind tunnel testing characterises the wing's aerodynamic performance, revealing significant deflection at high angles of attack due to the articulated skeletal structure. The wings demonstrate low power consumption, averaging only 5.1 W during morphing in experiments. Finally, we demonstrate the wing's robustness through outdoor flight experiments. The research findings provide insights into the potential of bioinspired designs for tailsitter UAVs and offer a promising avenue for future research in this field.

Grasping and manipulation are complex and demanding tasks, especially when executed in dynamic and unstructured environments. Typically, such tasks are executed by rigid articulated end-effectors, with a plethora of actuators that need sophisticated sensing and complex control laws to execute them efficiently. Soft robotics offers an alternative that allows for simplified execution of these demanding tasks, enabling the creation of robust, efficient, lightweight, and affordable solutions that are easy to control and operate. In this work, we introduce a new class of soft, kirigami-based robotic grippers, we study their post-contact behavior, and we investigate different cut patterns for their development. We follow an experimental approach in which several designs are proposed and employed in a series of grasping and force exertion tests to compare their capabilities and post-contact behavior. The results of such experiments indicate a clear relationship between degree of reconfiguration and grasping force, and provide key insights into the effect of the cut patterns in the performance of the designs. These findings are then used in the design process of an improved version of multi-layer, disposable kirigami grippers that are fabricated employing simple 3D printed layers and silicone rubber using the concept of Hybrid Deposition Manufacturing (HDM). A series of experimental results demonstrate that the proposed design and manufacturing methods can enable the creation of soft, kirigami-based grippers with superior grasping capabilities that can handle delicate, contaminated, and everyday life objects and can even be disposed off in an automated way (e.g., after handling hazardous materials, such as medical waste).

Soft robots are capable of effortlessly adapting to their environment using elastic materials that impart structural compliance into their designs, allowing them to execute complex tasks with minimal sensing and control. However, soft robots cannot exert high forces and can only handle low deformation forces. These characteristics typically limit their applicabil-ity to tasks that require delicate interactions. In this work, we present a mechanically programmable, variable stiffness, jamming actuator based on an articulated mesh structure. The proposed actuator can elastically bend when it is not activated but compresses to attain a pre-programmed shape that is determined by the mesh geometry of the multi-layer jamming architecture when pressure is applied to the silicone pouch containing it. Unlike traditional jamming structures the utilisation of the articulated mesh structure facilitates elastic deformations past the yield point when jammed. The actuator can become >27 times stiffer than its relaxed configuration when exposed to only 90 kPa pressure. We demonstrate the efficiency of this actuator by developing variable stiffness joints that can be used to create: i) underactuated, tendon driven robotic grippers and soft, disposable robotic grippers that exhibit increased dexterity and ii) wearable, affordable, lightweight elbow exoskeleton systems that can assist humans in holding heavy objects with minimal effort.

We introduce an efficient differentiable fluid simulator that can be integrated with deep neural networks as a part of layers for learning dynamics and solving control problems. It offers the capability to handle one-way coupling of fluids with rigid objects using a variational principle that naturally enforces necessary boundary conditions at the fluid-solid interface with sub-grid details. This simulator utilizes the adjoint method to efficiently compute the gradient for multiple time steps of fluid simulation with user defined objective functions. We demonstrate the effectiveness of our method for solving inverse and control problems on fluids with one-way coupled solids. Our method outperforms the previous gradient computations, state-of-the-art derivative-free optimization, and model-free reinforcement learning techniques by at least one order of magnitude.

We present a method for efficient differentiable simulation of articulated bodies. This enables integration of articulated body dynamics into deep learning frameworks, and gradient-based optimization of neural networks that operate on articulated bodies. We derive the gradients of the contact solver using spatial algebra and the adjoint method. Our approach is an order of magnitude faster than autodiff tools. By only saving the initial states throughout the simulation process, our method reduces memory requirements by two orders of magnitude. We demonstrate the utility of efficient differentiable dynamics for articulated bodies in a variety of applications. We show that reinforcement learning with articulated systems can be accelerated using gradients provided by our method. In applications to control and inverse problems, gradient-based optimization enabled by our work accelerates convergence by more than an order of magnitude.

We present a method for differentiable simulation of soft articulated bodies. Our work enables the integration of differentiable physical dynamics into gradient-based pipelines. We develop a top-down matrix assembly algorithm within Projective Dynamics and derive a generalized dry friction model for soft continuum using a new matrix splitting strategy. We derive a differentiable control framework for soft articulated bodies driven by muscles, joint torques, or pneumatic tubes. The experiments demonstrate that our designs make soft body simulation more stable and realistic compared to other frameworks. Our method accelerates the solution of system identification problems by more than an order of magnitude, and enables efficient gradient-based learning of motion control with soft robots.

Digital try-on systems for e-commerce have the potential to change people’s lives and provide notable economic benefits. However, their development is limited by practical constraints, such as accurate sizing of the body and realism of demonstrations. We enumerate three open challenges remaining for a complete and easy-to-use try-on system that recent advances in machine learning make increasingly tractable. For each, we describe the problem, introduce state-of-the-art approaches, and provide future directions.

Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. While notable progress has been made, the capabilities of differentiable physics solvers remain limited. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions. To accommodate objects with arbitrary geometry and topology, we adopt meshes as our representation and leverage the sparsity of contacts for scalable differentiable collision handling. Collisions are resolved in localized regions to minimize the number of optimization variables even when the number of simulated objects is high. We further accelerate implicit differentiation of optimization with nonlinear constraints. Experiments demonstrate that the presented framework requires up to two orders of magnitude less memory and computation in comparison to recent particle-based methods. We further validate the approach on inverse problems and control scenarios, where it outperforms derivative-free and model-free baselines by at least an order of magnitude.

We propose a scalable neural network framework to reconstruct the 3D mesh of a human body from multi-view images, in the subspace of the SMPL model. Use of multi-view images can significantly reduce the projection ambiguity of the problem, increasing the reconstruction accuracy of the 3D human body under clothing. Our experiments show that this method benefits from the synthetic dataset generated from our pipeline since it has good flexibility of variable control and can provide ground-truth for validation. Our method outperforms existing methods on real-world images, especially on shape estimations.

We propose a differentiable cloth simulator that can be embedded as a layer in deep neural networks. This approach provides an effective, robust framework for modeling cloth dynamics, self-collisions, and contacts. Due to the high dimensionality of the dynamical system in modeling cloth, traditional gradient computation for collision response can become impractical. To address this problem, we propose to compute the gradient directly using QR decomposition of a much smaller matrix. Experimental results indicate that our method can speed up backpropagation by two orders of magnitude. We demonstrate the presented approach on a number of inverse problems, including parameter estimation and motion control for cloth.

Abstract Cloth simulations, widely used in computer animation and apparel design, can be computationally expensive for real‐time applications. Some parallelization techniques have been proposed for visual simulation of cloth using CPU or GPU clusters and often rely on parallelization using spatial domain decomposition techniques that have a large communication overhead. In this paper, we propose a novel time‐domain parallelization technique that makes use of the two‐level mesh representation to resolve the time‐dependency issue and develop a practical algorithm to smooth the state transition from the corresponding coarse to fine meshes. A load estimation and a load balancing technique used in online partitioning are also proposed to maximize the performance acceleration. Our method achieves a nearly linear performance scaling on manycore clusters and outperforms spatial‐domain parallelization on a diverse set of benchmarks.

Image and video understanding enables better reconstruction of the physical world. Existing methods focus largely on geometry and visual appearance of the reconstructed scene. In this paper, we extend the frontier in image understanding and present a method to recover the material properties of cloth from a video. Previous cloth material recovery methods often require markers or complex experimental set-up to acquire physical properties, or are limited to certain types of images or videos. Our approach takes advantages of the appearance changes of the moving cloth to infer its physical properties. To extract information about the cloth, our method characterizes both the motion and the visual appearance of the cloth geometry. We apply the Convolutional Neural Network (CNN) and the Long Short Term Memory (LSTM) neural network to material recovery of cloth from videos. We also exploit simulated data to help statistical learning of mapping between the visual appearance and material type of the cloth. The effectiveness of our method is demonstrated via validation using both the simulated datasets and the real-life recorded videos.