To robustly handle objects, robots must perceive mechanical interactions through touch with sufficient richness. New tactile sensors leverage miniature cameras to provide dense measurements of these interactions, allowing for the extraction of material properties and frictional information. Among the plethora of solutions, retrographic sensing is popular for its ability to finely resolve the shape of the object being touched. These sensors use a reflective membrane, illuminated at a shallow angle by three RGB lights from which fine details of the surface can be recovered. However, these retrographic sensors are unable to detect the lateral displacement of the membrane and, therefore overlook frictional information, which is crucial for grasping and manipulation. Embedding and tracking opaque markers has been a makeshift solution, but these markers occlude the membrane and are difficult to manufacture. In this paper, we introduce ShadowTac, a tactile sensor that combines retrographic illumination with non-intrusive markers created by colored shadows. We patterned the retrographic surface with a dense array of submillimeter dimples, which are small enough not to obstruct the view yet cast shadows large enough to be visible to the camera. ShadowTac captures a dense image of both the normal displacement field with fine details and a precise lateral displacement field by tracking the markers. Additionally, our sensor is easy to manufacture, as the dimple pattern can simply be molded. We evaluated the measurement reliability of ShadowTac and its effectiveness in estimating the incipient slip of arbitrary objects. The dense measurement of both the normal and shear deformation that the sensor captures makes it ideal for tracking dynamic interactions between robotic fingertips and manipulated objects.

To gently grasp objects, robots need to balance generating enough friction yet avoiding too much force that could damage the object. In practice, the force regulation is challenging to implement since it requires knowledge of the friction coefficient, which can vary from object to object and even from grasp to grasp. Tactile sensing offers a window in the contact mechanics and provides information about friction. Notably touch can detect the precursor of the object slipping away from the grasp. To find this information, tactile sensors measure the deformation field of an artificial skin in both the normal and tangential direction. However, current approaches only react to slip and therefore react too late to perturbations. The object slips, inducing a failure of the grasp and damage. In this study, we introduce a method that uses machine-learning to anticipate slip by computing the so-called safety margin of the grasp. This safety margin represents the extra lateral force that maintains the contact away from the frictional limit. To find this value, we use a high-density camera-based tactile sensor to measure the 3D deformation of the surface via the movement of 82 colored markers. We trained a Convolutional Neural Network (CNN) to estimate the safety margin from the tactile images. Because it gives a distance to slip, the safety margin is a powerful metric for regulating grasp forces. As a testament of this effectiveness, we show that a simple proportional controller can robustly grasp a wide variety of objects. The results show that this control method outperforms slip detection methods, by reducing regrasp reaction times while decreasing grasping forces to 1-3 N.

Moving a slider to set the music volume or control the air conditioning is a familiar task that requires little attention. However, adjusting a virtual slider on a featureless touchscreen is much more demanding and can be dangerous in situations such as driving. Here, we study how a gradual tactile feedback, provided by a haptic touchscreen, can replace visual cues. As users adjust a setting with their finger, they feel a continuously changing texture, which spatial frequency correlates to the value of the setting. We demonstrate that, after training with visual and auditory feedback, users are able to adjust a setting on a haptic touchscreen without looking at the screen, thereby reducing visual distraction. Every learning strategy yielded similar performance, suggesting an amodal integration. This study shows that surface haptics can provide intuitive and precise tuning possibilities for tangible interfaces on touchscreens.

The only way to perceive a small object held between our fingers is to trust our sense of touch. Touch provides cues about the state of the contact even if its view is occluded by the finger. The interaction between the soft fingers and the surface reveals crucial information, such as the local shape of the object, that plays a central role in fine manipulation. In this work, we present a new spherical sensor that endows robots with a fine distributed sense of touch. This sensor is an evolution of our distributed tactile sensor that measures the dense 3-dimensional displacement field of an elastic membrane, using the subtractive color-mixing principle. We leverage a planar manufacturing process that enables the design and manufacturing of the functional features on a flat surface. The flat functional panels are then folded to create a spherical shape able to sense a wide variety of objects.The resulting 40mm-diameter spherical sensor has 77 measurement points, each of which gives an estimation of the local 3d displacement, normal and tangential to the surface. Each marker is built around 2 sets of colored patches placed at different depths. The relative motion and resulting hue of each marker, easily captured by an embedded RGB camera, provides a measurement of their 3d motion. To benchmark the sensor, we compared the measurements obtained while pressing the sensor on a curved surface with Hertz contact theory, a hallmark of contact mechanics. While the mechanics did strictly follow Hertz contact theory, using the shear and normal sensing, ChromaTouch can estimate the curvature of an object after a millimeter-size indentation of the sensor.

This paper presents the design of a bioinspired artificial fingertip that resembles the mechanical behavior of a human fingertip under conditions of both static deformation and high frequency excitation. The artificial fingertip is constructed around a deformable spherical membrane filled with a cellulose sponge, itself connected to a rigid structure that acts as a bone. Force-deformation characteristics and response to a transient mechanical perturbation are both shown to be in good qualitative agreement with those of a real finger. More importantly, the fingertip exhibits friction reduction when interacting with TPads (variable friction tactile displays based on transverse ultrasonic vibrations). Comparison with artificial fingertips that do not exhibit friction reduction suggests that mechanical damping characteristics play a key role in the amount of friction reduction achieved.

Human grip forces are automatically adjusted upon occurrence of an external disturbance experienced by an object that is held by a thumb and index finger. We investigated some of the cues that may be used by the brain to perform rapid grip restabilization. To this end we ask subjects to grip and hold an instrumented and actuated parallelepiped-shaped handle between the index finger and the thumb. Under computer control, the handle could be jerked from the still grip and could independently provided vibration of 250 or 100 Hz to the gripping fingers. We found that the latency of the motor corrective action was 139 ms on average, but when a vibrotactile stimulation was applied 50 ms before the application of the pulling force, the latency was reduced on average to 117 ms. The average latency of the conscious response to the vibrotactile stimuli was 230 ms, suggesting that vibrotactile stimulation was capable of influencing the reflex action.

We describe a mechanism for the delivery of haptic feedback to users of a simple game via the use of probabilistic inference. This not only enables the creation of dynamically changing game conditions but also a more adaptable, accessible and enjoyable haptic gaming environment for potential use by visually impaired users. The ViPong proof of concept application uses a mouse instrumented with a custom built lateral vibrator to enable a person to compete eyes-free in a game of pong using only haptic feedback linked to the position of the ball. A preliminary user study shows that it is possible to build eyes-free games using such a mechanism for haptic feedback generation. It also shows that there is some effect on the game play from varying levels of uncertainty injected into the algorithm, with participants performing less well under condition where the uncertainty added to the position of the game ball is high.

Grasping is one of the most common forms of dexterity. So far, most research has focused on slow-varying loads which can be resisted by anticipatory grip adjustments. There are common cases, however, when a rapid, unexpected increase in the load occurs and where the central nervous system must re-adjust the grip dynamically to prevent slippage. During such events, the central nervous system reactively updates the grip force to minimize further escape of an object. While existing theories postulates that the shear strain of the finger pads caused by the load force is a primary source of information for detecting a new load condition, vibrations induced by even minute object slip in the hand might more effectively signal the occurrence of unwanted movement of the object relatively to the hand. With the help of a high-sensitivity force sensor interposed in the load-path of a fast traction-creating device, we recorded the fluctuations of the force projected onto the fingertip when a rapid perturbation was applied to a grasped object. These fluctuations are indicative of slip. The results highlight the existence of a correlation between the amplitude of the vibrations and the grip force modulation, when textural features are present. The study provides promising evidence that the central nervous system exploits vibrations to detect the onset of unwanted movement of an object relatively to the hand to optimally scale the grip force in response to unexpected, rapid load variations.

Fluctuations of the frictional force arising from the stroke of a finger against flat and sinusoidal surfaces are studied. A custom-made high-resolution friction force sensor, able to resolve milli-newton forces, was used to record those fluctuations as well as the net, low-frequency components of the interaction force. Measurements show that the fluctuations of the sliding force are highly non-stationary. Despite their randomness, force spectra averages reveal regularities. With a smooth, flat, but not mirror-finish, surface the background noise follows a 1/f trend. Recordings made with pure-tone sinusoidal gratings reveal complexities in the interaction between a finger and a surface. The fundamental frequency is driven by the periodicity of the gratings and harmonics follow a non-integer power-law decay that suggests strong nonlinearities in the fingertip interaction. The results are consistent with the existence of a multiplicity of simultaneous and rapid stick-slip relaxation oscillations. Results have implications for high fidelity haptic rendering and biotribology.

The tactual scanning of five naturalistic textures was recorded with an apparatus that is capable of measuring the tangential interaction force with a high degree of temporal and spatial resolution. The resulting signal showed that the transformation from the geometry of a surface to the force of traction and, hence, to the skin deformation experienced by a finger is a highly nonlinear process. Participants were asked to identify simulated textures reproduced by stimulating their fingers with rapid, imposed lateral skin displacements as a function of net position. They performed the identification task with a high degree of success, yet not perfectly. The fact that the experimental conditions eliminated many aspects of the interaction, including low-frequency finger deformation, distributed information, as well as normal skin movements, shows that the nervous system is able to rely on only two cues: amplitude and spectral information. The examination of the “spatial spectrograms” of the imposed lateral skin displacement revealed that texture could be represented spatially, despite being sensed through time and that these spectrograms were distinctively organized into what could be called “spatial formants.” This finding led us to speculate that the mechanical properties of the finger enables spatial information to be used for perceptual purposes in humans with no distributed sensing, which is a principle that could be applied to robots.

This paper presents a novel tactile display device that exhibits some advantageous features for the blind: low-cost, lightweight, compactness and high portability. The prototype consists of an 8 times 8 arrangement of tactile pins actuated by shape memory alloys (SMAs) with 2.6 mm spatial resolution and 1 mm vertical excursion. Unlike existing devices, the full display is only 200 g weight and its compact dimensions make it easily carried by the user. A technical overview of the system is presented as well as preliminary results on tactile perception to evaluate its performance on information transmission