Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloudbased tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/.

Teledriving could serve as a practical solution for handling unforeseen situations in autonomous driving. However, the latency of transmission networks remains a prominent concern. Despite advancements like 5G networks, delays in remote driving scenes cannot be entirely eradicated, potentially leading to unwanted incidents. While a few attempts have been made to address this issue by predicting the future driving scenes, these efforts have been restricted in their ability to accurately foresee clear and relevant driving scenarios. This study presents a method to predict a latency-free future driving scene. Unlike prior approaches, our method incorporates the command signal of a remote driver into the prediction network, as well as the past driving video frames and vehicle status. As a result, we can accurately predict relevant and clear latency-free future driving scenes. A combination of convolutional long short-term memory (ConvLSTM) and generative adversarial networks (GAN) was utilized in a deep neural network to predict the future driving scenes based on latency. The dataset was gathered from on-road teledriving experiments, with a maximum vehicle speed of 53 km/h and a driving route length of approximately 1.3 km. The dataset used to train the deep neural network was gathered from on-road teledriving experiments. The proposed method can estimate the future driving scene for up to 0.5 s, surpassing the performance of both baseline video prediction methods and a method that does not utilize the input command of the driver.

When a robot is teleoperated, its operator control is based on transmitted images. Network limitations and/or a remote distance usually cause delays or interruptions of the image transmission, which is one of the reasons for the instability of teleoperation systems. In this article, we propose a high-update-rate image generation method using past low update image and current grip position and electrical motor current of gripper received by sensors during teleoperation via a conditional generative adversarial network. The main challenge is that such a network can generate current high-update-rate images from past low-update-rate one, the current high-update-rate grip force, and the grip angle. We equipped a robot gripper with a camera and a grip force sensor and collected a large data set of robot vision, grip force, and grip angle sequences; objects with deformation, including irregular deformation, and rigid objects were tested in the experiment to verify the possibility of high-update-rate image generation under various grip conditions. We found that the proposed network allows the generation of current images with high update rate.

Electronic skin (e-skin) is designed to mimic the comprehensive nature of human skin. Various advances in e-skin continue to drive the development of the multimodal tactile sensor technology on flexible and stretchable platforms. e-skin incorporates pressure, temperature, texture, photographic imaging, and other sensors as well as data acquisition and signal processing units formed on a soft substrate for humanoid robots, wearable devices, and health monitoring electronics that are the most critical applications of soft electronics. This artificial skin has developed very rapidly toward becoming real technology. However, the complex nature of e-skin technology presents significant challenges in terms of materials, devices, sophisticated integration methods, and interference-free data acquisition. These challenges range from functional materials, device architecture, pixel design, array structure, and data acquisition method to multimodal sensing performance with negligible interference. In this article, we present recent research trends and approaches in the field of flexible and stretchable multimodal sensors for e-skin focusing on the following aspects: 1) flexible and stretchable platforms; 2) operating principles and materials suitable for pressure, temperature, strain, photograph, and hairy sensor devices; 3) device and integration architectures, including multimodal single cells, three-axis tactile sensors, vertical-stacked sensor arrays, active matrix sensor arrays, and integration electronics; 4) reliable acquisition methods for various texture sensing and machine-learning algorithms for processing tactile sensing data; and 5) future outlook.

While interacting with mobile devices, users may press against touch screens and also exert tangential force to the display in a sliding manner. We seek to guide UI design based on the tangential force applied by a user to the surface of a hand-held device. A prototype of an interface using tangential force input was implemented utilizing a force sensitive layer and an elastic layer and used for the user experiment. We investigated user controllability to reach and maintain target force levels and considered the effects of hand pose and direction of force input. Our results imply no significant difference in performance when applying force holding the device in one hand and in two hands. We also observed that users have more physical and perceived loads when applying tangential force in the left-right direction compared to the up-down direction. Based on the experimental results, we discuss considerations for user interface applications of tangential-force-based interface.

To augment tangible information into current social network systems (SNSs), this paper proposes an algorithm that generates force data from the user's hand gestures. The input modality defined in this paper is based on gesture information in the form of an acceleration profile in that at times it describes the current status of a user better than text-based content. For the physical representation of a motion update, corresponding force vectors in a Cartesian coordinate system are calculated with the proposed linear equation. Given the experimental results, it is predicted that the proposed physical logging and rendering scheme will provide a physical experience in mediated social connections.

With an increase in the number of handicapped persons, there is a growing demand for human friendly interface as mobility aids. To meet this need, we have developed two interfaces with using shoulder and head motion for powered wheelchair control. To acquire proper wheelchair control instruction signal, workspaces of shoulder and head are analyzed by magnetic position sensor. Two interfaces are developed to meet four guidelines :human friendly design, easiness of wearability, intuitive drive function, and low cost. FSR (force sensing resistor) is used to measure changes in the shoulder and head motion. Interface's usefulness is verified by clinical experiment with six subjects who are spinal cord injured with C4 or C5.