In-hand manipulation is challenging in robotics due to the intricate contact dynamics and high degrees of control freedom. Precise manipulation with high accuracy often requires tactile perception, which adds further complexity to the system. Despite the challenges in perception and control, the rolling stick problem is an essential and practical motion primitive with many demanding industrial applications. This work aims to learn the high-resolution tactile dynamics of the rolling stick. Specifically, we try manipulating a small stick using the Allegro hand equipped with the Digit vision-based tactile sensor. The learning framework includes an action filtering module, tactile perception module, and learning with uncertainty module, all designed to operate in low data regimes. With only 2.3% amount of data and 5.7% model complexity of previous similar work, our learned contact dynamics model achieves better grasp stability, sub-millimeter precision, and promising zero-shot generalizability across novel objects. The proposed framework demonstrates the potential for precise in-hand manipulation with tactile feedback on real hardware. The project source code is available at: https://github.com/duyipai/Allegro_Digit. A video presentation is available here.

Perceiving the dynamical environment both visually and tactilely is crucial for the survival of animals, and therefore, is considered of importance in robotics research. Recently, there has been an increasing interest in vision-based tactile sensors due to their high sensing resolution and robustness to environmental changes. However, almost all vision-based tactile sensors make only partial use of the camera, specifically, only when contact occurs, and stay idle at other times, which results in a waste of the camera information bandwidth. In this paper, we propose a new visual-tactile dual-modality sensor called SpecTac, which can visually inspect the environment and make tactile observations. The main novelty of the sensor is the use of ultraviolet (UV) LEDs and randomly distributed UV fluorescent markers. When the LEDs are on, those markers will be bright and can easily be distinguished and tracked from the background. Besides, by controlling the on and off of the UV LEDs, due to the switchable visibility of those markers, the sensor will switch between visual and tactile sensing mode. The qualities of tactile and visual perception are evaluated quantitatively by force estimation, visual triangulation and visual feature matching. By combining both modalities into one compact sensor, the information from the camera is better utilized, and it is hoped that the sensor will achieve more flexibility in the motion of the robot arm, especially in tasks where the workspace is narrow.

Tactile sensing on human feet is crucial for motion control, however, has not been explored in robotic counterparts. This work is dedicated to endowing tactile sensing to legged robot’s feet and showing that a single-legged robot can be stabilized with only tactile sensing signals from its foot. We propose a robot leg with a novel vision-based tactile sensing foot system and implement a processing algorithm to extract contact information for feedback control in stabilizing tasks. A pipeline to convert images of the foot skin into high-level contact information using a deep learning framework is presented. The leg was quantitatively evaluated in a stabilization task on a tilting surface to show that the tactile foot was able to estimate both the surface tilting angle and the foot poses. Feasibility and effectiveness of the tactile system were investigated qualitatively in comparison with conventional single-legged robotic systems using inertia measurement units (IMU). Experiments demon-strate the capability of vision-based tactile sensors in assisting legged robots to maintain stability on unknown terrains and the potential for regulating more complex motions for humanoid robots.