Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale.

Hardness sensing is a valuable capability for a robot touch sensor. We describe a novel method of hardness sensing that does not require accurate control of contact conditions. A GelSight sensor is a tactile sensor that provides high resolution tactile images, which enables a robot to infer object properties such as geometry and fine texture, as well as contact force and slip conditions. The sensor is pressed on silicone samples by a human or a robot and we measure the sample hardness only with data from the sensor, without a separate force sensor and without precise knowledge of the contact trajectory. We describe the features that show object hardness. For hemispherical objects, we develop a model to measure the sample hardness, and the estimation error is about 4% in the range of 8 Shore 00 to 45 Shore A. With this technology, a robot is able to more easily infer the hardness of the touched objects, thereby improving its object recognition as well as manipulation strategy.

Artificial tactile sensing is still underdeveloped, especially in sensing shear and slip on a contact surface. For a robot hand to manually explore the environment or perform a manipulation task such as grasping, sensing of shear forces and detecting incipient slip is important. In this paper, we introduce a method of sensing the normal, shear and torsional load on the contact surface with a GelSight tactile sensor [1]. In addition, we demonstrate the detection of incipient slip. The method consists of inferring the state of the contact interface based on analysis of the sequence of images of GelSights elastomer medium, whose deformation under the external load indicates the conditions of contact. Results with a robot gripper like experimental setup show that the method is effective in detecting interactions with an object during stable grasp as well as at incipient slip. The method is also applicable to other optical based tactile sensors.

Robust manipulation and insertion of small parts can be challenging because of the small tolerances typically involved. The key to robust control of these kinds of manipulation interactions is accurate tracking and control of the parts involved. Typically, this is accomplished using visual servoing or force-based control. However, these approaches have drawbacks. Instead, we propose a new approach that uses tactile sensing to accurately localize the pose of a part grasped in the robot hand. Using a feature-based matching technique in conjunction with a newly developed tactile sensing technology known as GelSight that has much higher resolution than competing methods, we synthesize high-resolution height maps of object surfaces. As a result of these high-resolution tactile maps, we are able to localize small parts held in a robot hand very accurately. We quantify localization accuracy in benchtop experiments and experimentally demonstrate the practicality of the approach in the context of a small parts insertion problem.

A GelSight sensor is a tactile sensing device comprising a clear elastomeric pad covered with a reflective membrane, coupled with optics to measure the membrane's deformations. When the pad is pressed against an object's surface, the membrane changes shape in accord with mechanical and geometrical properties of the object. Since soft tissue is more compliant than hard tissue, one can detect an embedded lump by pressing the GelSight pad against the tissue surface and observing the hump that forms over the lump. We tested this system's sensitivity by constructing phantoms of soft rubber with hard embedded lumps. The system is quite sensitive; for example it could detect a 2mm lump at a depth of 5mm. The sensor was more sensitive than previous tactile lump detectors. It was also better than human observers using their fingertips. Such a capability could help in tumor screening, and could augment the sensory information available in telemedicine or minimally invasive surgery.

A novel thin film polydimethylsiloxane (PDMS) based flexible tactile micro sensor has been developed for contact traction distribution measurement between an indenter and an object (e.g. finger pad). The tactile sensor is a five layer device consisting of 2 layers of textured patterns sandwiched between 3 layers of PDMS. The textured patterns are protected by a thin layer of PDMS (25 microns) on each side and are separated by a thicker PDMS layer (100 microns). The sensor thus has a total thickness of 150 microns, much less than the thickness of the human fingertip skin. The entire device is fabricated bottom up on a silicon wafer using soft lithography and microfabrication techniques. The sensor is designed to be used with an imaging apparatus (in our case we use an Optical Coherence Tomography (OCT) apparatus). The deflection of the patterns due to an applied load is imaged by the OCT and is used to estimate the stress at the sensor and object interface by solving the stress inverse problem analytically. Results are presented to illustrate the use of the microsensor to estimate the contact traction distribution due to a spherical indenter of radius 500 μm.

The authors report six experiments on the human ability to discriminate and identify finger joint-angle positions using active motion. The PIP (proximal interphalangeal) joint of the index finger was examined in Exps. 1--3 and the MCP (metacarpophalangeal) joint in Exps. 4--6. In Exp. 1, the just noticeable difference (JND) of PIP joint-angle position was measured when the MCP joint was either fully extended or halfway bent. In Exp. 2, the JND of PIP joint-angle position as a function of PIP joint-angle reference position was measured when the PIP joint was almost fully extended, halfway bent, or almost fully flexed. In Exp. 3, the information transfer of PIP joint-angle position was estimated with the MCP joint in a fully extended position. In Exps. 4--6, the JND and the information transfer of MCP joint-angle position were studied with a similar experimental design. The results show that the JNDs of the PIP joint-angle position were roughly constant (2.5°−2.7°) independent of the PIP joint-angle reference position or the MCP joint-angle position used (Exps. 1 and 2). The JNDs of the MCP joint-angle position, however, increased with the flexion of both the PIP and MCP joints and ranged from 1.7° to 2.7° (Exps. 4 and 5). The information transfer of the PIP and MCP joint-angle position were similar, indicating 3--4 perfectly identifiable joint-angle positions for both joints (Exps. 3 and 6). The results provide the basic data needed for estimating, for example, the resolution of fingertip position during active free motion. They are compared to the results from previous studies on joint position, length, and thickness perception.

This paper proposes a tactile display device providing pin-array type tactile feedback and thermal feedback. The pin-array type tactile display is composed of a 6times5 pin-array that is actuated by 30 piezoelectric bimorphs. Micro shape and vibrotactile feedback can be generated by the device, and various planar distributed patterns can be displayed as can braille cell patterns. The thermal feedback device is composed of a thin film resistance temperature detector (RTD), a Peltier thermoelectric heat pump and a water cooling jacket. Users can discriminate among different materials by considering the temperature variation that can be sensed as they touch an object's surface. This paper also includes an experimental evaluation of the device to prove effectiveness of displaying textures. Material property discriminating evaluation was conducted using thermal feedback device that displays simulated temperature profile. To determine the relation between stimulated area and thermal perception sensitivity, a thermal perception experimental setup is developed and the experimental method is described

This paper presents a novel haptic mouse KAT (KAIST artificial touch) that can be used as a human-computer interface and offers the capability of displaying properties such as patterns, gratings and roughness. A small planar-distributed pin array is developed. The array can represent micro-scale shapes with various surfaces, such as gratings, grooves, patterns, shapes of icons, and Braille, and provides the user with cutaneous stimuli. Since the tactile display unit is small enough to be embedded into a computer mouse, we developed a new texture display mouse. The performance of its texture display capability was verified. In addition, two psychophysical experiments have been conducted in order to ascertain the influence of vibrotactile stimuli. The first experiment showed that vibrational stimuli could be effective in the perception of patterns while from the second experiment, the functional relation between perceived roughness and components of vibrotactile stimuli was obtained. The experimental results have been applied to the development of a test-bed program

Investigating virtual environments has become an increasingly interesting research topic for engineers, computer and cognitive scientists, and psychologists. Although there have been several recent studies focused on the development of multimodal virtual environments (VEs) to study human-machine interactions, less attention has been paid to human-human and human-machine interactions in shared virtual environments (SVEs), and to our knowledge, no attention paid at all to what extent the addition of haptic communication between people would contribute to the shared experience. We have developed a multimodal shared virtual environment and performed a set of experiments with human subjects to study the role of haptic feedback in collaborative tasks and whether haptic communication through force feedback can facilitate a sense of being and collaborating with a remote partner. The study concerns a scenario where two participants at remote sites must cooperate to perform a joint task in an SVE. The goals of the study are (1) to assess the impact of force feedback on task performance, (2) to better understand the role of haptic communication in human-human interactions, (3) to study the impact of touch on the subjective sense of collaborating with a human as reported by the participants based on what they could see and feel, and (4) to investigate if gender, personality, or emotional experiences of users can affect haptic communication in SVEs. The outcomes of this research can have a powerful impact on the development of next-generation human-computer interfaces and network protocols that integrate touch and force feedback technology into the internet, development of protocols and techniques for collaborative teleoperation such as hazardous material removal, space station.