Compliance is a useful parametrization of tactile information that humans often utilize in manipulation tasks. It can be used to inform low-level contact-rich actions or characterize objects at a high-level. In robotic manipulation, existing approaches to estimate compliance have struggled to generalize across both object shape and material. Using camera-based tactile sensors, proprioception, and force measurements, we present a novel approach to estimate object compliance as Young’s modulus E from parallel grasps. We evaluate our method over a novel dataset of 285 common objects, including a wide array of shapes and materials with Young’s moduli ranging from 5.0 kPa to 250 GPa. Combining analytical and data-driven approaches, we develop a hybrid system using a multi-tower neural network to analyze a sequence of tactile images from grasping. This system is shown to estimate the Young’s modulus of unseen objects within an order of magnitude at 74.2% accuracy across our dataset. This is an improvement over purely analytical and data-driven baselines which exhibit 28.9% and 65.0% accuracy respectively. Importantly, this estimation system performs irrespective of object geometry and demonstrates increased robustness across material types. Code is available on GitHub and collected data is available on HuggingFace.

To gently grasp objects, robots need to balance generating enough friction yet avoiding too much force that could damage the object. In practice, the force regulation is challenging to implement since it requires knowledge of the friction coefficient, which can vary from object to object and even from grasp to grasp. Tactile sensing offers a window in the contact mechanics and provides information about friction. Notably touch can detect the precursor of the object slipping away from the grasp. To find this information, tactile sensors measure the deformation field of an artificial skin in both the normal and tangential direction. However, current approaches only react to slip and therefore react too late to perturbations. The object slips, inducing a failure of the grasp and damage. In this study, we introduce a method that uses machine-learning to anticipate slip by computing the so-called safety margin of the grasp. This safety margin represents the extra lateral force that maintains the contact away from the frictional limit. To find this value, we use a high-density camera-based tactile sensor to measure the 3D deformation of the surface via the movement of 82 colored markers. We trained a Convolutional Neural Network (CNN) to estimate the safety margin from the tactile images. Because it gives a distance to slip, the safety margin is a powerful metric for regulating grasp forces. As a testament of this effectiveness, we show that a simple proportional controller can robustly grasp a wide variety of objects. The results show that this control method outperforms slip detection methods, by reducing regrasp reaction times while decreasing grasping forces to 1-3 N.

The only way to perceive a small object held between our fingers is to trust our sense of touch. Touch provides cues about the state of the contact even if its view is occluded by the finger. The interaction between the soft fingers and the surface reveals crucial information, such as the local shape of the object, that plays a central role in fine manipulation. In this work, we present a new spherical sensor that endows robots with a fine distributed sense of touch. This sensor is an evolution of our distributed tactile sensor that measures the dense 3-dimensional displacement field of an elastic membrane, using the subtractive color-mixing principle. We leverage a planar manufacturing process that enables the design and manufacturing of the functional features on a flat surface. The flat functional panels are then folded to create a spherical shape able to sense a wide variety of objects.The resulting 40mm-diameter spherical sensor has 77 measurement points, each of which gives an estimation of the local 3d displacement, normal and tangential to the surface. Each marker is built around 2 sets of colored patches placed at different depths. The relative motion and resulting hue of each marker, easily captured by an embedded RGB camera, provides a measurement of their 3d motion. To benchmark the sensor, we compared the measurements obtained while pressing the sensor on a curved surface with Hertz contact theory, a hallmark of contact mechanics. While the mechanics did strictly follow Hertz contact theory, using the shear and normal sensing, ChromaTouch can estimate the curvature of an object after a millimeter-size indentation of the sensor.