Recently introduced ControlNet has the ability to steer the text-driven image generation process with geometric input such as 2D human pose, or edge representations. While ControlNet provides control over the geometric form of the instances in the generated image, it lacks the capability to dictate the visual appearance of each instance. We present FineControlNet to provide fine control over each instance's appearance while maintaining the pose control capability. Specifically, we develop and demonstrate FineControlNet with geometric control via human pose images and appearance control via instance-level text prompts. The spatial alignment of 2D poses and instance-specific text prompts in latent space enables the fine control of multiple instances. We evaluate the performance of FineControlNet with rigorous comparison against state-of-the-art pose-conditioned text-to-image diffusion models. FineControlNet achieves superior performance in generating high quality images that follow instance-specific controls. We will release the code and the dataset.

A robot in a human-centric environment needs to account for the human’s intent and future motion in its task and motion planning to ensure safe and effective operation. This requires symbolic reasoning about probable future actions and the ability to tie these actions to specific locations in the physical environment. While one can train behavioral models capable of predicting human motion from past activities, this approach requires large amounts of data to achieve acceptable long-horizon predictions. More importantly, the resulting models are constrained to specific data formats and modalities. Moreover, connecting predictions from such models to the environment at hand to ensure the applicability of these predictions is an unsolved problem. We present a system that utilizes a Large Language Model (LLM) to infer a human’s next actions from a range of modalities without fine-tuning. A novel aspect of our system that is critical to robotics applications is that it links the predicted actions to specific locations in a semantic map of the environment. Our method leverages the fact that LLMs, trained on a vast corpus of text describing typical human behaviors, encode substantial world knowledge, including probable sequences of human actions and activities. We demonstrate how these localized activity predictions can be incorporated in a human-aware task planner for an assistive robot to reduce the occurrences of undesirable human-robot interactions by 29.2% on average.

As robots begin to move from structured industrial environments to the real world, they must be equipped to not only safely interact with the environment, but also reason about how to leverage contact to perform tasks. In this work, we develop a modeling and motion planning framework for continuum robots that accounts for contact anywhere along the robot. We first present an analytical model for continuum manipulators under contact and discuss the ideal choice of generalized coordinates given properties of the manipulator and task specifications. We then demonstrate the utility of our model by developing a motion planning framework that can solve a diverse set of tasks. We apply our framework to end effector path planning for a soft arm in an obstacle-rich environment, and grasp planning for soft robotic grippers, where contact can happen anywhere on the arm or gripper. Finally, we verify the utility of our model and planning framework by planning a grasp with a desired contact force for a soft antipodal gripper and testing this grasp in a hardware demonstration. Overall, our model and planning approach further enhance soft and continuum robots where they already excel: utilizing contact with the world to achieve their goals with a gentle touch.

In-hand manipulation is challenging for soft robotic hands, especially in the real world where robots encounter a variety of object sizes and shapes. As such, the role of the palm is crucial, providing stabilizing contact to objects during grasping and manipulation, and controlling the position of objects with respect to the fingertips. We demonstrate an actuated palm capable of enhancing the in-hand manipulation capabilities of a soft hand by better-utilizing limited finger dexterity. With a combination of physical and virtual experiments, we explore the effects of palm diameter and height on in-hand manipulation performance over a variety of object shapes and sizes, and three key manipulation primitive motions. The results of these experiments show that maintaining manipulation capabilities over a large range of object sizes requires the palm’s diameter to decrease as a function of its height to prevent interference between the fingers and palm. Based on these insights, we design an actuated palm mechanism that achieves the desired relationship between palm height and diameter using one actuated degree of freedom. Finally, we show that this adjustable palm enables the hand to manipulate a larger range of object sizes and aspect ratios, and its utility is demonstrated in a mid-air shelving in-hand manipulation task.

Engineers and scientists often rely on their intuition and experience when designing soft robotic systems. The development of performant controllers and motion plans for these systems commonly requires time-consuming iterations on hardware. We present the SoMo (Soft Motion) toolkit, a software framework that makes it easy to instantiate and control typical continuum manipulators in an accurate physics simulator. SoMo introduces a standardized and human-readable description format for continuum manipulators. It leverages this description format and the Bullet physics engine to enable fast and accurate simulations of soft and soft-rigid hybrid robots in environments with complex contact interactions. This allows users to vary design and control parameters across simulations with minimal effort. We compare the capabilities of SoMo to other physics simulators and highlight the benefits and accuracy of SoMo by demonstrating the agreement between simulation and real-world experiments on several examples; these include an in-hand manipulation task with continuum fingers, an automated exploration of how to design soft fingers for precision grasping, and a brief snake locomotion study. Overall, SoMo provides an accessible way for designers of soft robotic hardware and control systems to gain access to a simulation-accelerated workflow.

The need for robotic hands capable of gentle in-hand manipulation is growing rapidly as robots enter the real world. In this work, we show that the arrangement of digits in a soft robotic hand has a strong effect on in-hand manipulation capabilities. Introducing task-based performance metrics which quantify the range of motion, repeatability, and accuracy of in-hand manipulation tasks, we investigate hand designs with finger arrangements ranging from axisymmetric-circular to anthropomorphic. Using an open-source soft robot simulator, the effect of object size and aspect ratio on the in-hand manipulation performance is studied for a variety of finger arrangements, and findings are validated using a physical hardware platform. We found that the ideal finger arrangement is task-dependent; anthropomorphic arrangements excel at lateral translations, and axisymmetric arrangements are best suited for rotations. The aspect ratio of the object also has a strong effect on in-hand manipulation, with anthropomorphic designs performing best on objects of high aspect ratio, and axisymmetric arrangements doing well on objects of low aspect ratio. These findings are further confirmed in a real-world task with delicate pastries, where gentle in-hand manipulation is critical. Overall, our results suggest that active control of digit arrangement is necessary for soft robotic hands to maximize in-hand manipulation capabilities with arbitrary objects.

Soft strain sensors have been explored as an unobtrusive approach for wearable motion tracking. However, accurate tracking of multi degree-of-freedom (DOF) noncyclic joint movements remains a challenge. This paper presents a soft sensing shirt for tracking shoulder kinematics of both cyclic and random arm movements in 3 DOFs: adduction/abduction, horizontal flexion/extension, and internal/external rotation. The sensing shirt consists of 8 textile-based capacitive strain sensors sewn around the shoulder joint that communicate to a customized readout electronics board through sewn micro-coaxial cables. An optimized sensor design includes passive shielding and demonstrates high linearity and low hysteresis, making it suitable for wearable motion tracking. In a study with a single human subject, we evaluated the tracking capability of the integrated shirt in comparison with a ground truth optical motion capture system. An ensemble-based regression algorithm was implemented in post-processing to estimate joint angles and angular velocities from the strain sensor data. Results demonstrated root mean square errors (RMSEs) less than 4.5° for joint angle estimation and normalized root mean square errors (NRMSEs) less than 4% for joint velocity estimation. Furthermore, we applied a recursive feature elimination (RFE)-based sensor selection analysis to down select the number of sensors for future shirt designs. This sensor selection analysis found that 5 sensors out of 8 were sufficient to generate comparable accuracies.

Domains where supervised models are deployed often come with task-specific constraints, such as prior expert knowledge on the ground-truth function, or desiderata like safety and fairness. We introduce a novel probabilistic framework for reasoning with such constraints and formulate a prior that enables us to effectively incorporate them into Bayesian neural networks (BNNs), including a variant that can be amortized over tasks. The resulting Output-Constrained BNN (OC-BNN) is fully consistent with the Bayesian framework for uncertainty quantification and is amenable to black-box inference. Unlike typical BNN inference in uninterpretable parameter space, OC-BNNs widen the range of functional knowledge that can be incorporated, especially for model users without expertise in machine learning. We demonstrate the efficacy of OC-BNNs on real-world datasets, spanning multiple domains such as healthcare, criminal justice, and credit scoring.

Insect-scale flying robots are currently unable to carry the power source and sensor suite required for autonomous operation. To overcome this challenge, we developed and experimentally verified a non-linear damping model of actuation-limited flapping-wing vehicles with passively rotating wing hinges. In agreement with studies on the wing dynamics of honey bees, we found that the optimal angle of the passive wing hinge in mid-stroke is about 70 ° rather than 45-50 ° as previously assumed. We further identified a narrow actuation force window in which the occurrence of a sharp resonance can be used to achieve both higher lift and efficiency. The findings from our model informed design changes to the Harvard Dual-Actuator Robobee, which resulted in a 130% increase in mean lift from ~140mg to 320mg (with a vehicle mass increase of only 5 - 8%), along with a corresponding expected payload increase of 330 - 470% (30 - 40mg to 170mg). The power consumption only increased by ~55%, making the new prototype 50% more efficient at lift production. Our model provides a greater understanding of the dynamics of this complex system, and the resulting lift and efficiency improvements are expected to bring insect-scale flying robots closer to autonomy.