Tactile sensors play a crucial role in enabling robots to interact effectively and safely with objects in everyday tasks. In particular, visuotactile sensors have seen increasing usage in two and three-fingered grippers due to their high-quality feedback. However, a significant gap remains in the development of sensors suitable for humanoid robots, especially five-fingered dexterous hands. One reason is because of the challenges in designing and manufacturing sensors that are compact in size. In this paper, we propose HumanFT, a multimodal visuotactile sensor that replicates the shape and functionality of a human fingertip. To bridge the gap between human and robotic tactile sensing, our sensor features real-time force measurements, high-frequency vibration detection, and overtemperature alerts. To achieve this, we developed a suite of fabrication techniques for a new type of elastomer optimized for force propagation and temperature sensing. Besides, our sensor integrates circuits capable of sensing pressure and vibration. These capabilities have been validated through experiments. The proposed design is simple and cost-effective to fabricate. We believe HumanFT can enhance humanoid robots' perception by capturing and interpreting multimodal tactile information.

Robot skill acquisition processes driven by reinforcement learning often rely on simulations to efficiently generate large-scale interaction data. However, the absence of simulation models for tactile sensors has hindered the use of tactile sensing in such skill learning processes, limiting the development of effective policies driven by tactile perception. To bridge this gap, we present TwinTac, a system that combines the design of a physical tactile sensor with its digital twin model. Our hardware sensor is designed for high sensitivity and a wide measurement range, enabling high quality sensing data essential for object interaction tasks. Building upon the hardware sensor, we develop the digital twin model using a real-to-sim approach. This involves collecting synchronized cross-domain data, including finite element method results and the physical sensor’s outputs, and then training neural networks to map simulated data to real sensor responses. Through experimental evaluation, we characterized the sensitivity of the physical sensor and demonstrated the consistency of the digital twin in replicating the physical sensor’s output. Furthermore, by conducting an object classification task, we showed that simulation data generated by our digital twin sensor can effectively augment real-world data, leading to improved accuracy. These results highlight TwinTac’s potential to bridge the gap in cross-domain learning tasks.

Tactile perception is crucial for robots to interact effectively with their environments, particularly in cluttered settings or when visual sensing is unavailable. However, a major limitation is the insufficient coverage of tactile sensors on current robots, which makes navigating cluttered spaces challenging due to the lack of capability to detect collisions. This limitation also hinders the use of teleoperation systems in such spaces by reducing the human operator’s situational awareness. To address this issue, this paper proposes the Touch-Linked Sleeve (TLS), a haptic mapping system that redirects contact on robot arms to human skin. The system consists of a tactile skin for contact detection and a haptic sleeve that enables human operators to experience telepresented contact. By establishing a transparent mapping between the robot’s tactile skin and the user’s haptic sleeve, operators can intuitively sense contacts from the robot’s perspective. To evaluate the system’s effectiveness, we conducted experiments demonstrating the functionality of both the tactile skin and the haptic sleeve. Moreover, we performed human studies using a virtual reality robot teleoperation interface to simulate navigation and manipulation in a cluttered scenario. The results indicate that the proposed system enhances perceptual transparency during object grasping tasks, leading to improved task completion times, fewer collisions, and improved overall usability.

Endowing the curved surfaces of rounded vision-based tactile fingers is essential for dexterous robotic manipulation, as they offer more sufficient contact with the environment. However, current rounded designs are constrained by a low sensing frequency (30–60 Hz) and the need for recalibration when adapting to new sensors due to the reliance on multi-channel captures, which hinders their performance in dynamic robotic tasks and large-scale deployment. In this work, we introduce R-Tac0, a low-cost rounded VBTS engineered for high-resolution and high-speed perception. The key innovation is a monochrome vision-based sensing principle: utilizing a black-and-white camera to capture the reflection properties of the compound rounded elastomer under monochromatic illumination. This single-channel imaging significantly reduces data volume and simplifies computational complexity, enabling 120 Hz tactile perception. A lightweight neural network can calibrate the sensor to achieve a depth reconstruction accuracy of 0.169 mm per pixel, while exhibiting surprisingly good transferability to new sensors. In experiments, we demonstrate the advantages of R-Tac0’s rounded design by evaluating its performance under different contact angles, its high-frequency perception in slip detection, and its effectiveness in robotic dynamic pose estimation.

Trajectory prediction for individual vehicles has emerged as a vital component in Internet of Things (IoT)-based traffic management applications, inducing various control strategies for alleviating traffic congestion. This study focuses on a novel topic in this field, i.e., making joint predictions for the next location and travel time. Based on principles of vehicle mobility, we learn vehicle trajectories as discrete events in the spatiotemporal dimension and propose a neural temporal point process, named TrajTPP. This model employs two attention mechanisms to learn spatial and temporal dependencies, respectively, and a novel recurrent structure is proposed to integrate spatiotemporal features. Meanwhile, a gated residual attentive network (GRAN) is also designed to combine these learned dynamic features with static travel information. Then, the intensity-free learning strategy is employed to make probabilistic forecasting for the next travel times, and we develop a prior transition probability to involve historical travel behaviors in location predictions. Beyond the conventional prediction task, we design a sampling strategy to simulate vehicle mobilities by TrajTPP. Experiments from license plate recognition data in Changsha, China, demonstrate that our model outperforms advanced baselines, and sampling results provide evidence of its ability to accurately simulate vehicle mobilities. Moreover, its impressive accuracy on the latest next-location prediction benchmark is also listed in the Appendix.

Handling fragile objects requires minimally invasive interaction skills in order to avoid any permanent deformation, alternation or damages. Such need is often required in tactile exploration tasks. In this paper, we propose an innovative whisker manipulator (COMPlacent), which is designed to accomplish tactile exploration with minimum intrusiveness. The design is inspired by biological whiskers observed in animals, where whiskers are used as means of tactile exploration in an analogous way as fingers are. Artificial whiskers are compliant but robust, which mitigates contact forces by bending or conforming to the object surface. The intrusiveness is further reduced by reactive control, which is implemented based on tactile sensors and actuators installed on each whisker. This allows the whisker to be retracted from the object surface, so that the energy transferred by contacts is minimized. The tactile sensor is designed to be ultrasensitive, which allows it to gather contact information with high fidelity. By modeling contact pressure as a time-series signal, a machine learning framework is leveraged to discriminate object properties including shape and texture. Evaluation experiments were conducted on real objects, which successfully demonstrates object classification at an accuracy of 97.3%, and texture discrimination accuracy of 92.1%.

Traversing through a tilted narrow gap is previously an intractable task for reinforcement learning mainly due to two challenges. First, searching feasible trajectories is not trivial because the goal behind the gap is difficult to reach. Second, the error tolerance after Sim2Real is low due to the relatively high speed in comparison to the gap's narrow dimensions. This problem is aggravated by the intractability of collecting real-world data due to the risk of collision damage. In this brief, we propose an end-to-end reinforcement learning framework that solves this task successfully by addressing both problems. To search for dynamically feasible flight trajectories, we use a curriculum learning to guide the agent toward the sparse reward behind the obstacle. To tackle the Sim2Real problem, we propose a Sim2Real framework that can transfer control commands to a real quadrotor without using real flight data. To the best of our knowledge, our brief is the first work that accomplishes successful gap traversing task purely using deep reinforcement learning.

Robots can be used to mitigate risks in unsafe and austere settings. In recent years, explosive ordnance disposal robots have reduced the technician's time-on-target, and thus, reduce the direct risk of exposure. This article focuses on the study and development of innovative techniques as the foundational work for a new robot platform. The proposed system includes an organic electrochemical transistor device to detect the existence of explosive residues, and lead to decisions for safe-removal progress. Taurus' surgical gripper facilitates object tactile exploration, and manipulation with control precision to the millimeter range. The highly sensitive triboelectric tactile sensor could reduce intrusiveness during contact, and mitigate the risk of detonation. Haptic devices and visual displays are used to convey important signals, in order to improve the situational awareness of the teleoperator. A machine learning classifier can be used to assist the user to identify objects from tactile sampling. The integration of these methodologies allows for a sensitive approach to concealed objects that are only accessible through tactile sensing.

Robotic exploration under uncertain environments is challenging when optical information is not available. In this article, we propose an autonomous solution of exploring an unknown task space based on tactile sensing alone. We first designed a whisker sensor based on MEMS barometer devices. This sensor can acquire contact information by interacting with the environment nonintrusively. This sensor is accompanied by a planning technique to generate exploration trajectories by using mere tactile perception. This technique relies on a hybrid policy for tactile exploration, which includes a proactive informative path planner for object searching, and a reactive Hopf oscillator for contour tracing. Results indicate that the hybrid exploration policy can increase the efficiency of object discovery. Last, scene understanding was facilitated by segmenting objects and classification. A classifier was developed to recognize the object categories based on the geometric features collected by the whisker sensor. Such an approach demonstrates the whisker sensor, together with the tactile intelligence, can provide sufficiently discriminative features to distinguish objects.

Humans have the innate ability to recognize new objects just by looking at sketches of them (also referred as to proto-type images). Similarly, prototypical images can be used as an effective visual representations of unseen classes to tackle few-shot learning (FSL) tasks. Our main goal is to recognize unseen hand signs (gestures) traffic-signs, and corporate-logos, by having their iconographic images or prototypes. Previous works proposed to utilize variational prototypical-encoders (VPE) to address FSL problems. While VPE learns an image-to-image translation task efficiently, we discovered that its performance is significantly hampered by the so-called hubness problem and it fails to regulate the representations in the latent space. Hence, we propose a new model (VPE++) that inherently reduces hubness and incorporates contrastive and multi-task losses to increase the discriminative ability of FSL models. Results show that the VPE++ approach can generalize better to the unseen classes and can achieve superior accuracies on logos, traffic signs, and hand gestures datasets as compared to the state-of-the-art.

Three dimensional (3D) object recognition is becoming a key desired capability for many computer vision systems such as autonomous vehicles, service robots and surveillance drones to operate more effectively in unstructured environments. These real-time systems require effective classification methods that are robust to various sampling resolutions, noisy measurements, and unconstrained pose configurations. Previous research has shown that points' sparsity, rotation and positional inherent variance can lead to a significant drop in the performance of point cloud based classification techniques. However, neither of them is sufficiently robust to multifactorial variance and significant sparsity. In this regard, we propose a novel approach for 3D classification that can simultaneously achieve invariance towards rotation, positional shift, scaling, and is robust to point sparsity. To this end, we introduce a new feature that utilizes graph structure of point clouds, which can be learned end-to-end with our proposed neural network to acquire a robust latent representation of the 3D object. We show that such latent representations can significantly improve the performance of object classification and retrieval tasks when points are sparse. Further, we show that our approach outperforms PointNet and 3DmFV by 35.0% and 28.1% respectively in ModelNet 40 classification tasks using sparse point clouds of only 16 points under arbitrary SO(3) rotation.