Artificial tactile sensing is still underdeveloped, especially in sensing shear and slip on a contact surface. For a robot hand to manually explore the environment or perform a manipulation task such as grasping, sensing of shear forces and detecting incipient slip is important. In this paper, we introduce a method of sensing the normal, shear and torsional load on the contact surface with a GelSight tactile sensor [1]. In addition, we demonstrate the detection of incipient slip. The method consists of inferring the state of the contact interface based on analysis of the sequence of images of GelSights elastomer medium, whose deformation under the external load indicates the conditions of contact. Results with a robot gripper like experimental setup show that the method is effective in detecting interactions with an object during stable grasp as well as at incipient slip. The method is also applicable to other optical based tactile sensors.

Robust manipulation and insertion of small parts can be challenging because of the small tolerances typically involved. The key to robust control of these kinds of manipulation interactions is accurate tracking and control of the parts involved. Typically, this is accomplished using visual servoing or force-based control. However, these approaches have drawbacks. Instead, we propose a new approach that uses tactile sensing to accurately localize the pose of a part grasped in the robot hand. Using a feature-based matching technique in conjunction with a newly developed tactile sensing technology known as GelSight that has much higher resolution than competing methods, we synthesize high-resolution height maps of object surfaces. As a result of these high-resolution tactile maps, we are able to localize small parts held in a robot hand very accurately. We quantify localization accuracy in benchtop experiments and experimentally demonstrate the practicality of the approach in the context of a small parts insertion problem.

Sensing surface textures by touch is a valuable capability for robots. Until recently it was difficult to build a compliant sensor with high sensitivity and high resolution. The GelSight sensor is compliant and offers sensitivity and resolution exceeding that of the human fingertips. This opens the possibility of measuring and recognizing highly detailed surface textures. The GelSight sensor, when pressed against a surface, delivers a height map. This can be treated as an image, and processed using the tools of visual texture analysis. We have devised a simple yet effective texture recognition system based on local binary patterns, and enhanced it by the use of a multi-scale pyramid and a Hellinger distance metric. We built a database with 40 classes of tactile textures using materials such as fabric, wood, and sandpaper. Our system can correctly categorize materials from this database with high accuracy. This suggests that the GelSight sensor can be useful for material recognition by robots.

A GelSight sensor is a tactile sensing device comprising a clear elastomeric pad covered with a reflective membrane, coupled with optics to measure the membrane's deformations. When the pad is pressed against an object's surface, the membrane changes shape in accord with mechanical and geometrical properties of the object. Since soft tissue is more compliant than hard tissue, one can detect an embedded lump by pressing the GelSight pad against the tissue surface and observing the hump that forms over the lump. We tested this system's sensitivity by constructing phantoms of soft rubber with hard embedded lumps. The system is quite sensitive; for example it could detect a 2mm lump at a depth of 5mm. The sensor was more sensitive than previous tactile lump detectors. It was also better than human observers using their fingertips. Such a capability could help in tumor screening, and could augment the sensory information available in telemedicine or minimally invasive surgery.