Robotic grasping of 3D deformable objects is critical for real-world applications such as food handling and robotic surgery. Unlike rigid and articulated objects, 3D deformable objects have infinite degrees of freedom. Fully defining their state requires 3D deformation and stress fields, which are exceptionally difficult to analytically compute or experimentally measure. Thus, evaluating grasp candidates for grasp planning typically requires accurate, but slow 3D finite element method (FEM) simulation. Sampling-based grasp planning is often impractical, as it requires evaluation of a large number of grasp candidates. Gradient-based grasp planning can be more efficient, but requires a differentiable model to synthesize optimal grasps from initial candidates. Differentiable FEM simulators may fill this role, but are typically no faster than standard FEM. In this work, we propose learning a predictive graph neural network (GNN), DefGraspNets, to act as our differentiable model. We train DefGraspNets to predict 3D stress and deformation fields based on FEM-based grasp simulations. DefGraspNets not only runs up to 1500x faster than the FEM simulator, but also enables fast gradient-based grasp optimization over 3D stress and deformation metrics. We design DefGraspNets to align with real-world grasp planning practices and demonstrate generalization across multiple test sets, including real-world experiments.

Accurately simulating whether an object will be lifted securely or dropped during grasping is a longstanding Sim2Real challenge. Soft compliant jaw tips are almost universally used with parallel-jaw robot grippers due to their ability to increase contact area and friction between the jaws and the object to be manipulated. However, interactions between the compliant surfaces and rigid objects are notoriously difficult to model. We introduce IPC-GraspSim, a novel grasp simulator that extends Incremental Potential Contact (IPC) - a highly accurate collision + deformation model developed in 2020 for computer graphics. IPC-GraspSim models both the dynamics and the deformation of compliant jaw tips to reduce Sim2Real gap for robot grasping. We evaluate IPC-GraspSim using a set of 2,000 physical grasps across 16 adversarial objects where analytic models perform poorly. In comparison to both analytic quasistatic contact models (soft point contact, REACH, 6DFC) and dynamic grasp simulators (Isaac Gym with FleX), results suggest IPC-GraspSim can predict robustness with higher precision and recall (F1 = 0.85). IPC-GraspSim increases F1 score by 0.03 to 0.20 over analytic baselines and 0.09 over Isaac Gym, at a cost of 8000x and 1.5x more compute time, respectively. All data, code, videos, and supplementary material are available at https://sites.google.com/berkeley.edu/ipcgraspsim.

The automated cleaning of surfaces such as furniture, bathroom sinks, and even human bodies is challenging due to the three-dimensional nature of their geometries. Yet, enabling robots to effectively and safely perform these tasks would not only reduce user efforts spent on household cleaning chores, but would also alleviate the strenuous workload of caretakers as the elderly population continues to grow at an unprecedented rate. In this work, we unify the applications of wiping objects and bathing humans as a general contour-following problem. To this end, we utilize a depth camera-based soft tactile sensor to extract the contact geometries and force-correlated measures during interaction between the robot and the target object or body part, and design a general contour-following controller that not only maintains contact with the target throughout the cleaning process, but also regulates the amount of force applied. Our system enables successful cleaning of pipes, shelving, and even human limbs and torsos without the need for data-driven methods such as deep learning, upon which the majority of existing works have relied.

The safety and comfort of drivers have been improved over the decades as a result of our broadened understanding of driver modeling and behavior prediction. Despite these remarkable advances in autonomous and interactive systems, there is a significant lack of approaches that consider the passengers and the vehicle as components of a dynamical vibro-acoustical system. Sound in vehicles is not only informative of the state of the vehicle and the environment, but can also critically affect the driver's performance, attention, and comfort. This paper aims to investigate the interplay between the perceived sounds of a vehicle and psychoacoustic annoyance (PA) metrics. Our goal is to create an intelligent agent that would act to improve driving pleasantness through acoustic-driven learning. To tackle the problem of choosing the correct actions to reduce the acoustic annoyance, the paper presents a method based on reinforcement learning that learns from the environment, i.e., the vehicle interior. The method actively changes the state inside the vehicle (e.g., closing or opening the window and choosing the cruise speed) in order to minimize acoustic annoyance experienced by the driver. The results of this work, performed using the GTA V simulator, showed that the trained agent successfully learned to take the correct actions to reduce PA metrics. The paper also present to the community a new multi-modal dataset composed of several rides on a real vehicle and an in-depth analysis of the influence of vehicle's signal on the acoustic annoyance.

If robots and humans are to coexist and cooperate in society, it would be useful for robots to be able to engage in tactile interactions. Touch is an intuitive communication tool as well as a fundamental method by which we assist each other physically. Tactile abilities are challenging to engineer in robots, since both mechanical safety and sensory intelligence are imperative. Existing work reveals a trade-off between these principles- tactile interfaces that are high in resolution are not easily adapted to human-sized geometries, nor are they generally compliant enough to guarantee safety. On the other hand, soft tactile interfaces deliver intrinsically safe mechanical properties, but their non-linear characteristics render them difficult for use in timely sensing and control. We propose a robotic system that is equipped with a completely soft and therefore safe tactile interface that is large enough to interact with human upper limbs, while producing high resolution tactile sensory readings via depth camera imaging of the soft interface. We present and validate a data-driven model that maps point cloud data to contact forces, and verify its efficacy by demonstrating two real-world applications. In particular, the robot is able to react to a human finger's pokes and change its pose based on the tactile input. In addition, we also demonstrate that the robot can act as an assistive device that dynamically supports and follows a human forearm from underneath.

Deploying robot learning frameworks in unconstrained environments requires robustness and tractability. We must not only equip the robot with a sufficient range of sensing capabilities, but also provide training data in a sample-efficient manner. To this end, we identify and address a need specifically in robot learning from demonstration (LfD) literature to account for not only end-effector pose and wrench signals, but also tactile signals for contact. While traditional pose and wrench signals have proven to be sufficient for robots to learn basic position and force-control behaviors, they are inherently too constraining for the learning of general manipulation tasks. In particular, useful manipulation tasks often rely on the geometry of the contact interaction. To explore the value of geometry-based tactile signals, we utilize a LfD framework built upon hidden Markov models and Gaussian mixture regression, adapt it to our robotic system equipped with a soft tactile sensor, and validate its performance with an edge-following task and a manipulation task involving different object geometries.

As the demand for robotic applications in unconstrained and dynamic environments rises, so does the benefit of advancing the state of the art in soft robotic technologies. However, the complex capabilities of soft robots elicited by their high-dimensional, non-linear characteristics simultaneously yield difficult challenges in control and sensing. Moreover, embedding tactile sensing capabilities in soft materials is often expensive and difficult to fabricate. In recent years, however, the invention of small-scale depth-sensing cameras introduced a promising channel for soft tactile sensor design. In this work, we propose a novel soft device inspired by the human fingertip that not only utilizes a small depth camera as the perception mechanism, but also possesses compliance-modulating capabilities. We demonstrate its ability to accurately estimate contact regions upon interaction with an external obstacle, and show that the estimation sensitivity can be modulated via internal fluid states. In addition, we determine an empirical model of the device's force-deformation characteristics under simplifying assumptions, and validate its performance with real-time force matching control experiments.

Despite the impressive progress being made in autonomous vehicles, human drivers will remain ubiquitous in the imminent years. Therefore, intelligent hybrid vehicular systems must be aware of the interactions between humans and the environment (e.g., sound, vibration, speed, etc.). In this paper, we evaluate the effect of acoustic annoyance on drivers in a real-world driving study. We found significant differences in driving styles elicited by annoying acoustics and present an online classifier that uses onboard inertial measurement unit measurements to distinguish whether a driver is annoyed with 77% accuracy. Moreover, we directly measured the forces applied on the passenger with a pressure mat lined on the car seat, and empirically confirm that our proposed passenger dynamics model is reasonable. However, due to our acoustically induced driving styles not being polarizing enough, we were unable to show that passengers' self-reported ride comfort changed with acoustic annoyance.

Currently, providing teachers with detailed feedback about their classroom discourse strategies requires highly trained observers to hand code transcripts of classroom recordings to identify talk moves and/or one-on-one expert coaching. Both approaches are time-consuming and expensive, require considerable human expertise, and do not scale to large numbers of teachers. We are currently developing an innovative application, the TalkBack application, a new type of teacher learning environment based on the automated analysis of classroom recordings. The TalkBack application will utilize a big data infrastructure for managing and analyzing classroom recordings, including an embedded automated talk move classifier. The application will provide teachers with a detailed record of the discourse strategies used in their lessons. A central premise of our research is that this type of personalized, automated feedback can dramatically enhance teacher learning and support improvements in their instruction.The project will exemplify how next-generation repositories of classroom recordings can be architected to support large-scale research by enabling automated analyses based on machine learning models.