Recent advances in machine learning have driven a step-change in robot perception with modalities such as vision, where large amounts of training data are readily available or cheap to collect. However, in tactile perception, the relatively high cost of data collection still largely impedes the adoption of such data-driven learning solutions. In this article, we introduce TactGen, a novel, cross-modal framework to tackle this challenge. In particular, using a two-step data generation pipeline, we leverage easily accessible vision data to synthesise artificial tactile data for downstream classifier training. Specifically, we use readily collected video data of objects of interest to efficiently learn neural radiance field (NeRF) representations. The NeRF models are then used to render red–green–blue-depth (RGBD) images from any desired vantage points. In the second stage, the RGBD images are translated into corresponding tactile images typically generated by camera-based tactile sensors using a conditional generative adversarial network (cGAN). The cGAN model is itself trained with a large set of visuo-tactile images collected in simulation, and can be transferred into the real world without fine-tuning or additional data collection. We extensively validate this approach in the context of tactile object classification, showing that it effectively reduces data collection time by a factor of 20 while achieving similar performance to training a classifier on manually collected real data.

For several tasks, ranging from manipulation to inspection, it is beneficial for robots to localize a target object in their surroundings. In this paper, we propose an approach that utilizes coarse point clouds obtained from miniaturized VL53L5CX Time-of-Flight (ToF) sensors (tiny LiDARs) to localize a target object in the robot’s workspace. We first conduct an experimental campaign to calibrate the dependency of sensor readings on relative range and orientation to targets. We then propose a probabilistic sensor model, which we validate in an object pose estimation task using a Particle Filter (PF). The results show that the proposed sensor model improves the performance of the localization of the target object with respect to two baselines: one that assumes measurements are free from uncertainty and one in which the confidence is provided by the sensor datasheet.

Tracking the motion of humans or objects in a robot’s surroundings is essential to improve safe robot motions and reactions. In this work, we present an approach for scene flow estimation from low-density and noisy point clouds acquired from miniaturised Time-of-Flight (ToF) sensors distributed across the robot’s body. The proposed method clusters points from consecutive frames and applies the Iterative Closest Point (ICP) algorithm to estimate a dense motion flow, with additional steps introduced to mitigate the impact of sensor noise and low-density data points. Specifically, we employ a fitness-based classification to distinguish between stationary and moving points and an inlier removal strategy to refine geometric correspondences.The proposed approach is validated in an experimental setup where 24 ToF are used to estimate the velocity of an object moving at different controlled speeds. Experimental results show that the method consistently approximates the direction of the motion and its magnitude with an error which is in line with sensor noise.

Tactile gesture recognition systems play a crucial role in Human-Robot Interaction (HRI) by enabling intuitive communication between humans and robots. The literature mainly addresses this problem by applying machine learning techniques to classify sequences of tactile images encoding the pressure distribution generated when executing the gestures. However, some gestures can be hard to differentiate based on the information provided by tactile images alone.In this paper, we present a simple yet effective way to improve the accuracy of a gesture recognition classifier. Our approach focuses solely on processing the tactile images used as input by the classifier. In particular, we propose to explicitly highlight the dynamics of the contact in the tactile image by computing the dense optical flow. This additional information makes it easier to distinguish between gestures that produce similar tactile images but exhibit different contact dynamics. We validate the proposed approach in a tactile gesture recognition task, showing that a classifier trained on tactile images augmented with optical flow information achieved a 9% improvement in gesture classification accuracy compared to one trained on standard tactile images.

Recent advances in machine learning have driven a step-change in robot perception with modalities such as vision, where large amounts of training data are readily available or cheap to collect. However, in tactile perception, the relatively high cost of data collection still largely impedes the adoption of such data-driven learning solutions. In this article, we introduce TactGen, a novel, cross-modal framework to tackle this challenge. In particular, using a two-step data generation pipeline, we leverage easily accessible vision data to synthesise artificial tactile data for downstream classifier training. Specifically, we use readily collected video data of objects of interest to efficiently learn neural radiance field (NeRF) representations. The NeRF models are then used to render red–green–blue-depth (RGBD) images from any desired vantage points. In the second stage, the RGBD images are translated into corresponding tactile images typically generated by camera-based tactile sensors using a conditional generative adversarial network (cGAN). The cGAN model is itself trained with a large set of visuo-tactile images collected in simulation, and can be transferred into the real world without fine-tuning or additional data collection. We extensively validate this approach in the context of tactile object classification, showing that it effectively reduces data collection time by a factor of 20 while achieving similar performance to training a classifier on manually collected real data.

Robots performing tasks in challenging environments must be supported by control or planning algorithms that exploit sensor feedback to effectively plan the robot’s actions. In this paper, we propose a reactive control law that simultaneously utilizes proximity and tactile feedback to perform a pick-and-place task in an unknown and cluttered environment. Specifically, the presented solution leverages proximity sensing obtained from distributed Time of Flight (ToF) sensors to avoid collision when this does not interfere with the pick-and-place task. Safety is guaranteed by a higher-priority task using tactile feedback that reduces contact forces when a collision occurs. Additionally, we compare the effectiveness of this control scheme with a collision detection and reaction scheme based solely on tactile sensing. Our results demonstrate that the proposed approach reduces the collisions with the environment and the task execution time of the pick-and-place operation.

Intubation is essential for ventilating critically ill patients and involves precise maneuvering of a laryngoscope to place an endotracheal tube (ETT). However, training for this procedure is fraught with challenges. Traditional methods, relying on manikins or training with a single sensing modality, fail to adequately convey important interaction information. Such challenge is heightened in pediatric intubation due to anatomical differences that demand greater precision. Furthermore, integrating multiple sensing modalities into a laryngoscope without changing its size presents a significant design challenge, critical for maintaining realistic training scenarios. To overcome these obstacles, we developed a sensorized laryngoscope system equipped with a force-torque sensor, a 9-axis inertial measurement unit (IMU), and tactile sensors. This system, validated in a preliminary user study, provides online feedback on angles, forces, and grip strength through an online feedback GUI. Adopting a learning-by-demonstration approach with both experts and novices, the initial validation confirmed the system’s potential, paving the way for expanded trials with more participants.

Recent developments in vision-based tactile sensing offer a simple means to enable robots to perceive touch interactions. However, existing sensors are primarily designed for small-scale applications like robotic hands, lacking research on their integration for large-sized robot bodies that can be leveraged for safe human-robot interactions. This paper explores the utilization of the previously-developed vision-based tactile sensing link (called TacLink) with soft skin as a safety control mechanism, which can serve as an alternative to conventional rigid robot links and impact observers. We characterize the behavior of a robot integrated with the soft TacLink in response to collisions, particularly employing a reactive control strategy. The controller is primarily driven by tactile force information acquired from the soft TacLink sensor through a data-driven sim2real learning method. Compared with a standard rigid link, the results obtained from collision experiments also confirm the advantages of our "soft" solution in impact resilience and in facilitating controls that are difficult to achieve with a stiff robot body. This study can act as a benchmark for assessing the efficiency of soft tactile-sensitive skins in reactive collision responses and open new safety standards for soft skin-based collaborative robots in human-robot interaction scenarios.

The Peg-In-Hole (PiH) task performed under un-certain conditions still represents a challenge for autonomous robots. When the peg is not rigidly connected to the robot end-effector, the external forces generated by peg-environment interactions can change the in-hand pose of the peg. This aspect must be taken into account when performing the insertion. This paper deals with this problem and proposes an insertion strategy driven by tactile feedback. In particular, we consider holding the peg using a parallel gripper equipped with tactile sensors, whose measurements are processed to capture in-hand rotations of the peg pose. This information is fed back to the robot controller and used to compensate for changes in the peg orientation and end-point position occurring during the task execution. The approach is validated on a real robot using a two-finger gripper equipped with two capacitive-based tactile sensor arrays hosting 20 tactile elements each. We show that the proposed method achieves an insertion success rate of 38/40 with a 0.1 mm clearance between the peg and hole.

Tactile data processing or classification is commonly performed using tactile images, i.e. two-dimensional representation of the applied contact. When tactile sensors cover the whole robot body, their relative spatial relations change depending on the robot posture. If the applied contact involves two adjacent links, the current relations among the tactile elements must be considered in order to generate a tactile image that preserves the contact shape. The goal of this paper is to propose a method for creating tactile images from pressure measurement acquired from a large area tactile system, where the relative displacement among the sensors fixed to different links change according to the robot posture. The proposed approach is experimentally validated using a Baxter robot equipped with distributed tactile sensors.

This paper deals with the problem of estimating the pose of tactile elements (i.e. taxels) composing a robotic skin covering the whole body of a robot. This problem arises when a robot skin technology has to be integrated into an already existing robotic platform. To date, the integration process is done by hand and it is not possible to predict where the sensor will be placed on the robot body. This paper presents a novel approach based on a RGB-D camera and exploiting the motion capabilities of the robot for activating the skin sensors. The method uses the measurements of the camera to reconstruct the unknown robot body outer shape and to compute how the area can be touched by the robot. The taxels responses and the related contact centroids are used for estimating the position of the sensors. Our method is based on few assumptions and is a step towards a calibration procedure that can be executed autonomously by a robot. Experiments performed on the Baxter robotic platform demonstrate the effectiveness of the presented approach obtaining an average position error less than 2mm.

This paper deals with the problem of using the tactile feedback generated by a robotic skin for discriminating a human hand touch from a generic contact. Humans understand collaboration intentions through different sensing modalities such as vision, hearing and touch. Among them, a physical interaction is mainly used for demonstrating or correcting a kind of motion and is usually started by touching with the hands the other human body. Until recently, it was difficult to perform the same in human-robot cooperation due to the lack of large-scale tactile systems functionally similar to a human skin. Our approach consists in transforming measurements of sensors distributed on the robot body into a convenient 2D representation of the contact shape, i.e., a contact image, then applying image classification techniques in order to discriminate a human touch from unexpected collisions. Experiments have been performed on a robotic skin composed of 768 pressure sensors integrated on a Baxter robot forearm. More than 1800 contact images have been generated from 43 different persons for training and testing two machine learning algorithms: Bag of Visual Words and Convolutional Neural Networks. The experimental results show that both approaches are valid, obtaining a classification accuracy higher than 96%.

This paper presents a novel approach for recognizing a human hand touch by processing pressure measurements generated by a robotic skin. Physical cooperation among humans is mainly based on the sense of touch and usually starts with hand contacts. If a robot can distinguish a human touch from a generic contact, the human-robot cooperation can be more natural and effective. The proposed approach consists in transforming the sensor pressure measurements distributed on the robot surface into a convenient 2D representation of the contact shape, i.e., a contact image. The image-based representation of contacts allows facing the problem of human touch classification by applying machine learning methods already developed for image classification. The experiments have been performed using a robotic skin, composed of 768 tactile elements, placed on a Baxter robot forearm. The contact classification has been performed using a Convolutional Neural Network obtaining an accuracy higher than 97% experimentally validating the proposed approach.