Recent advances in machine learning have driven a step-change in robot perception with modalities such as vision, where large amounts of training data are readily available or cheap to collect. However, in tactile perception, the relatively high cost of data collection still largely impedes the adoption of such data-driven learning solutions. In this article, we introduce TactGen, a novel, cross-modal framework to tackle this challenge. In particular, using a two-step data generation pipeline, we leverage easily accessible vision data to synthesise artificial tactile data for downstream classifier training. Specifically, we use readily collected video data of objects of interest to efficiently learn neural radiance field (NeRF) representations. The NeRF models are then used to render red–green–blue-depth (RGBD) images from any desired vantage points. In the second stage, the RGBD images are translated into corresponding tactile images typically generated by camera-based tactile sensors using a conditional generative adversarial network (cGAN). The cGAN model is itself trained with a large set of visuo-tactile images collected in simulation, and can be transferred into the real world without fine-tuning or additional data collection. We extensively validate this approach in the context of tactile object classification, showing that it effectively reduces data collection time by a factor of 20 while achieving similar performance to training a classifier on manually collected real data.

For several tasks, ranging from manipulation to inspection, it is beneficial for robots to localize a target object in their surroundings. In this paper, we propose an approach that utilizes coarse point clouds obtained from miniaturized VL53L5CX Time-of-Flight (ToF) sensors (tiny LiDARs) to localize a target object in the robot’s workspace. We first conduct an experimental campaign to calibrate the dependency of sensor readings on relative range and orientation to targets. We then propose a probabilistic sensor model, which we validate in an object pose estimation task using a Particle Filter (PF). The results show that the proposed sensor model improves the performance of the localization of the target object with respect to two baselines: one that assumes measurements are free from uncertainty and one in which the confidence is provided by the sensor datasheet.

Tracking the motion of humans or objects in a robot’s surroundings is essential to improve safe robot motions and reactions. In this work, we present an approach for scene flow estimation from low-density and noisy point clouds acquired from miniaturised Time-of-Flight (ToF) sensors distributed across the robot’s body. The proposed method clusters points from consecutive frames and applies the Iterative Closest Point (ICP) algorithm to estimate a dense motion flow, with additional steps introduced to mitigate the impact of sensor noise and low-density data points. Specifically, we employ a fitness-based classification to distinguish between stationary and moving points and an inlier removal strategy to refine geometric correspondences.The proposed approach is validated in an experimental setup where 24 ToF are used to estimate the velocity of an object moving at different controlled speeds. Experimental results show that the method consistently approximates the direction of the motion and its magnitude with an error which is in line with sensor noise.

Tactile gesture recognition systems play a crucial role in Human-Robot Interaction (HRI) by enabling intuitive communication between humans and robots. The literature mainly addresses this problem by applying machine learning techniques to classify sequences of tactile images encoding the pressure distribution generated when executing the gestures. However, some gestures can be hard to differentiate based on the information provided by tactile images alone.In this paper, we present a simple yet effective way to improve the accuracy of a gesture recognition classifier. Our approach focuses solely on processing the tactile images used as input by the classifier. In particular, we propose to explicitly highlight the dynamics of the contact in the tactile image by computing the dense optical flow. This additional information makes it easier to distinguish between gestures that produce similar tactile images but exhibit different contact dynamics. We validate the proposed approach in a tactile gesture recognition task, showing that a classifier trained on tactile images augmented with optical flow information achieved a 9% improvement in gesture classification accuracy compared to one trained on standard tactile images.

Recent advances in machine learning have driven a step-change in robot perception with modalities such as vision, where large amounts of training data are readily available or cheap to collect. However, in tactile perception, the relatively high cost of data collection still largely impedes the adoption of such data-driven learning solutions. In this article, we introduce TactGen, a novel, cross-modal framework to tackle this challenge. In particular, using a two-step data generation pipeline, we leverage easily accessible vision data to synthesise artificial tactile data for downstream classifier training. Specifically, we use readily collected video data of objects of interest to efficiently learn neural radiance field (NeRF) representations. The NeRF models are then used to render red–green–blue-depth (RGBD) images from any desired vantage points. In the second stage, the RGBD images are translated into corresponding tactile images typically generated by camera-based tactile sensors using a conditional generative adversarial network (cGAN). The cGAN model is itself trained with a large set of visuo-tactile images collected in simulation, and can be transferred into the real world without fine-tuning or additional data collection. We extensively validate this approach in the context of tactile object classification, showing that it effectively reduces data collection time by a factor of 20 while achieving similar performance to training a classifier on manually collected real data.

Robots performing tasks in challenging environments must be supported by control or planning algorithms that exploit sensor feedback to effectively plan the robot’s actions. In this paper, we propose a reactive control law that simultaneously utilizes proximity and tactile feedback to perform a pick-and-place task in an unknown and cluttered environment. Specifically, the presented solution leverages proximity sensing obtained from distributed Time of Flight (ToF) sensors to avoid collision when this does not interfere with the pick-and-place task. Safety is guaranteed by a higher-priority task using tactile feedback that reduces contact forces when a collision occurs. Additionally, we compare the effectiveness of this control scheme with a collision detection and reaction scheme based solely on tactile sensing. Our results demonstrate that the proposed approach reduces the collisions with the environment and the task execution time of the pick-and-place operation.

Intubation is essential for ventilating critically ill patients and involves precise maneuvering of a laryngoscope to place an endotracheal tube (ETT). However, training for this procedure is fraught with challenges. Traditional methods, relying on manikins or training with a single sensing modality, fail to adequately convey important interaction information. Such challenge is heightened in pediatric intubation due to anatomical differences that demand greater precision. Furthermore, integrating multiple sensing modalities into a laryngoscope without changing its size presents a significant design challenge, critical for maintaining realistic training scenarios. To overcome these obstacles, we developed a sensorized laryngoscope system equipped with a force-torque sensor, a 9-axis inertial measurement unit (IMU), and tactile sensors. This system, validated in a preliminary user study, provides online feedback on angles, forces, and grip strength through an online feedback GUI. Adopting a learning-by-demonstration approach with both experts and novices, the initial validation confirmed the system’s potential, paving the way for expanded trials with more participants.

Recent developments in vision-based tactile sensing offer a simple means to enable robots to perceive touch interactions. However, existing sensors are primarily designed for small-scale applications like robotic hands, lacking research on their integration for large-sized robot bodies that can be leveraged for safe human-robot interactions. This paper explores the utilization of the previously-developed vision-based tactile sensing link (called TacLink) with soft skin as a safety control mechanism, which can serve as an alternative to conventional rigid robot links and impact observers. We characterize the behavior of a robot integrated with the soft TacLink in response to collisions, particularly employing a reactive control strategy. The controller is primarily driven by tactile force information acquired from the soft TacLink sensor through a data-driven sim2real learning method. Compared with a standard rigid link, the results obtained from collision experiments also confirm the advantages of our "soft" solution in impact resilience and in facilitating controls that are difficult to achieve with a stiff robot body. This study can act as a benchmark for assessing the efficiency of soft tactile-sensitive skins in reactive collision responses and open new safety standards for soft skin-based collaborative robots in human-robot interaction scenarios.

The Peg-In-Hole (PiH) task performed under un-certain conditions still represents a challenge for autonomous robots. When the peg is not rigidly connected to the robot end-effector, the external forces generated by peg-environment interactions can change the in-hand pose of the peg. This aspect must be taken into account when performing the insertion. This paper deals with this problem and proposes an insertion strategy driven by tactile feedback. In particular, we consider holding the peg using a parallel gripper equipped with tactile sensors, whose measurements are processed to capture in-hand rotations of the peg pose. This information is fed back to the robot controller and used to compensate for changes in the peg orientation and end-point position occurring during the task execution. The approach is validated on a real robot using a two-finger gripper equipped with two capacitive-based tactile sensor arrays hosting 20 tactile elements each. We show that the proposed method achieves an insertion success rate of 38/40 with a 0.1 mm clearance between the peg and hole.

Given the importance of affective touch in human interactions, technology designers are increasingly attempting to bring this modality to the core of interactive technology. Advances in haptics and touch-sensing technology have been critical to fostering interest in this area. In this survey, we review how affective touch is investigated to enhance and support the human experience with or through technology. We explore this question across three different research areas to highlight their epistemology, main findings, and the challenges that persist. First, we review affective touch technology through the human–computer interaction literature to understand how it has been applied to the mediation of human–human interaction and its roles in other human interactions particularly with oneself, augmented objects/media, and affect-aware devices. We further highlight the datasets and methods that have been investigated for automatic detection and interpretation of affective touch in this area. In addition, we discuss the modalities of affective touch expressions in both humans and technology in these interactions. Second, we separately review how affective touch has been explored in human–robot and real-human–virtual-human interactions where the technical challenges encountered and the types of experience aimed at are different. We conclude with a discussion of the gaps and challenges that emerge from the review to steer research in directions that are critical for advancing affective touch technology and recognition systems. In our discussion, we also raise ethical issues that should be considered for responsible innovation in this growing area.

Robotic phantoms enable advanced physical examination training before using human patients. In this article, we present an abdominal phantom for palpation training with controllable stiffness liver nodules that can also sense palpation forces. The coupled sensing and actuation approach is achieved by pneumatic control of positive-granular jammed nodules for tunable stiffness. Soft sensing is done using the variation of internal pressure of the nodules under external forces. This article makes original contributions to extend the linear region of the neo-Hookean characteristic of the mechanical behavior of the nodules by 140% compared to no-jamming conditions and to propose a method using the organ level controllable nodules as sensors to estimate palpation position and force with a root-mean-square error of 4% and 6.5%, respectively. Compared to conventional soft sensors, the method allows the phantom to sense with no interference to the simulated physiological conditions when providing quantified feedback to trainees, and to enable training following current bare-hand examination protocols without the need to wear data gloves to collect data.

To match the ever increasing standards of fresh products, and the need to reduce waste, we devise an alternative to the destructive and highly variable fruit ripeness estimation by a penetrometer. We propose a fully automatic method to assess the ripeness of mango which is non-destructive, allows the user to test multiple surface areas with a single touch and is capable of dissociating between ripe and non-ripe fruits. A custom-made gripper equipped with a capacitive tactile sensor array is used to palpate the fruit. The ripeness is estimated as mango stiffness extracted through a simplified spring model. We test the framework on a set of 25 mangoes of the Keitt variety, and compare the results to penetrometer measurements. We show it is possible to correctly classify 88% of the mango without removing the skin of the fruit. The method can be a valuable substitute for non-destructive fruit ripeness testing. To the authors knowledge, this is the first robotics ripeness estimation system based on capacitive tactile sensing technology.

In this paper a novel multi-modal tactile sensor is presented, featuring a matrix of capacitive pressure sensors, a microphone for acoustic measurements and proximity and ambient light sensor. The sensor is fully embedded and can be easily integrated at mechanical and electrical levels with industrial grippers. Tactile sensing design has been put on the same level of additional requirements, usually overlooked in tactile sensor research, such as the mechanical interface, cable harness and robustness against continuous and repetitive operations, just to name but a few. The performances of the different sensing modalities have been assessed in a test rig for tactile sensors. Experiments have been performed in order to show the capabilities of the sensor for implementing tactile based industrial gripper control and tactile based fabric classification.

In this paper, the description of a sensorized glove that has been developed to perform experiments in robot-based manipulation of clothes and objects is reported. The glove embeds a capacitive tactile sensing technology that has been designed in the past few years. The glove is used to provide an estimate of the expected tactile feedback related to involved forces and contact areas during common manipulation tasks. This information will be used in order to design a robot gripper for cloth manipulation.

Even though the sense of touch is crucial for humans, most humanoid robots lack tactile sensing. While a large number of sensing technologies exist, it is not trivial to incorporate them into a robot. We have developed a compliant “skin” for humanoids that integrates a distributed pressure sensor based on capacitive technology. The skin is modular and can be deployed on nonflat surfaces. Each module scans locally a limited number of tactile-sensing elements and sends the data through a serial bus. This is a critical advantage as it reduces the number of wires. The resulting system is compact and has been successfully integrated into three different humanoid robots. We have performed tests that show that the sensor has favorable characteristics and implemented algorithms to compensate the hysteresis and drift of the sensor. Experiments with the humanoid robot iCub prove that the sensors can be used to grasp unmodeled, fragile objects.