To achieve a desired grasping posture (including object position and orientation), multi-finger motions need to be conducted according to the the current touch state. Specifically, when subtle changes happen during correcting the object state, not only proprioception but also tactile information from the entire hand can be beneficial. However, switching motions with high-DOFs of multiple fingers and abundant tactile information is still challenging. In this study, we propose a loss function with constraints of touch states and an attention mechanism for focusing on important modalities depending on the touch states. The policy model is AE-LSTM which consists of Autoencoder (AE) which compresses abundant tactile information and Long Short-Term Memory (LSTM) which switches the motion depending on the touch states. Motion for cap-opening was chosen as a target task which consists of sub tasks of sliding an object and opening its cap. As a result, the proposed method achieved the best success rates with a variety of objects for real time cap-opening manipulation. Furthermore, we could confirm that the proposed model acquired the features of each subtask and attention on specific modalities.

Multi-fingered hands can be suitable for stable object manipulation. Furthermore, abundant tactile information can be acquired with multi-fingered hands, useful to recognize the object’s properties, which is beneficial to adapt the motion to the object. However, generating dexterous manipulation motions with multi-fingered hands with high density tactile sensors is challenging due to complex touch states. Hence, tasks that conventionally require a high level of active tactile sensing simultaneously with motion generation, such as pulling in the hand while recognizing the posture of an object are difficult to accomplish. In this letter, we propose a novel deep predictive learning approach using Vision-Transformer (ViT) and Long-Short Term Memory (LSTM). The ViT’s attention mechanism can spatially focus on specific fingers represented by distributed 3-axis tactile sensors (uSkin). The LSTM can preserve long time-series information of the manipulation which can realize changing the desired motion according to the initial touching position and orientation for the target object. Results showed that the ViT-LSTM is effective in performing adaptive finger movements according to the properties of the object, i.e. its hardness and relative posture.

Intelligent robots are expected to do manipulation tasks relying on real-time sensing feedback. Especially, tactile sensing plays a more and more important role in precise manipulation tasks. For example, a 1 mm error while inserting a USB stick, which is hard to perceive visually, will result in a failed insertion or even break the USB stick. In this paper, to estimate and compensate residual position uncertainties during robotic insertion tasks, an exploration motion is introduced to acquire environment information by tactile sensing and a state-of-the-art transformer-based neural network is proposed to estimate the error distance from long-duration tactile sensing data. Our system is trained on over 2000 insertion trials with basic geometry shaped 3D printed objects. Without any prior knowledge, we achieve an 85% insertion success rate with average 5 attempts on 4 unseen daily objects relying only on tactile feedback acquired from our proposed exploratory motion. It is noteworthy that our designed exploration motion can provide insightful information about extrinsic contact information and our proposed learning model exceeds previous baselines in extracting useful information regarding the contact interaction between the grasped object and the environment.

Multifingered robot hands can be extremely effective in physically exploring and recognizing objects, especially if they are extensively covered with distributed tactile sensors. Convolutional neural networks (CNNs) have been proven successful in processing high dimensional data, such as camera images, and are, therefore, very well suited to analyze distributed tactile information as well. However, a major challenge is to organize tactile inputs coming from different locations on the hand in a coherent structure that could leverage the computational properties of the CNN. Therefore, we introduce a morphology-specific CNN (MS-CNN), in which hierarchical convolutional layers are formed following the physical configuration of the tactile sensors on the robot. We equipped a four-fingered Allegro robot hand with several uSkin tactile sensors; overall, the hand is covered with 240 sensitive elements, each one measuring three-axis contact force. The MS-CNN layers process the tactile data hierarchically: at the level of small local clusters first, then each finger, and then the entire hand. We show experimentally that, after training, the robot hand can successfully recognize objects by a single touch, with a recognition rate of over 95%. Interestingly, the learned MS-CNN representation transfers well to novel tasks: by adding a limited amount of data about new objects, the network can recognize nine types of physical properties.

Skill transfer from humans to robots is challenging. Presently, many researchers focus on capturing only position or joint angle data from humans to teach the robots. Even though this approach has yielded impressive results for grasping applications, reconstructing motion for object handling or fine manipulation from a human hand to a robot hand has been sparsely explored. Humans use tactile feedback to adjust their motion to various objects, but capturing and reproducing the applied forces is an open research question. In this paper we introduce a wearable fingertip tactile sensor, which captures the distributed 3-axis force vectors on the fingertip. The fingertip tactile sensor is interchangeable between the human hand and the robot hand, meaning that it can also be assembled to fit on a robot hand such as the Allegro hand. This paper presents the structural aspects of the sensor as well as the methodology and approach used to design, manufacture, and calibrate the sensor. The sensor is able to measure forces accurately with a mean absolute error of 0.21, 0.16, and 0.44 Newtons in X, Y, and Z directions, respectively.

Detecting the onset/ongoing of slip, i.e. if a grasped object is slipping or will slip from the gripper while being lifted, is crucial. Conventionally, it is regarded as a tactile sensing related problem. However, recently multi-modal robotic learning has become popular and is expected to boost the performance. In this paper we propose a novel CNN-TCN model to fuse tactile and visual information for detecting the onset/ongoing of slip. In our experiments, two uSkin tactile sensors and one Realsense435i camera are used. Data is collected by randomly grasping and lifting 35 daily objects 1050 times in total. Furthermore, we compare our CNN-TCN model with the widely used CNN-LSTM model. As a result, our proposed model achieves a 88.75% detection accuracy and outperforms the CNN-LSTM model combined with different pretrained vision networks.

Recently, tactile sensing has attracted great interest for robotic manipulation. Predicting if a grasp will be stable or not, i.e. if the grasped object will drop out of the gripper while being lifted, can aid robust robotic grasping. Previous methods paid equal attention to all regions of the tactile data matrix or all time-steps in the tactile sequence, which may include irrelevant or redundant information. In this paper, we propose to equip Convolutional Neural Networks with spatial-channel and temporal attention mechanisms (SCT attention CNN) to predict future grasp stability. To the best of our knowledge, this is the first time to use attention mechanisms for predicting grasp stability only relying on tactile information. We implement our experiments with 52 daily objects. Moreover, we compare different spatio-temporal models and attention mechanisms as an empirical study. We found a significant accuracy improvement of up to 5% when using SCT attention. We believe that attention mechanisms can also improve the performance of other tactile learning tasks in the future, such as slip detection and hardness perception.

Object picking with two-fingered grippers is widely used in practice. However, the deformability and slipperiness of the target object still remain a challenge, and not resolving them might lead to breaking or dropping of the grasped objects. To prevent such instances, tactile sensing plays an important role because it can directly detect even the subtle changes that occur during grasping. Mechanoreceptors in the human skin detect such events by the change in the skin shape and/or vibration. Using a similar approach, a combined deformation and slip detection system using a distributed 3axis tactile information with various time-scales is proposed. Specifically, the tactile information includes the z-axis data, which denotes the deformation of the skin perpendicular to the finger’s surface and the x- and y-axes, which measure deformations tangential to the surface. The perpendicular and tangential tactile information are used to determine the deformation and slip, respectively. The system is based on a multilayer perceptron (MLP) that outputs detection results from a 3-axis tactile information. Results showed that, the perpendicular and tangential tactile information with an appropriate timescale were effective for deformation and slip detection with over 89% and 95% recognition rates, respectively, measured for 40 different objects. Moreover, 195 out of 200 real-time untrained grasping states were successful detected. Finally, 10 untrained objects were successfully picked.

The use of tactile information is one of the most important factors for achieving stable in-grasp manipulation. Especially with low-cost robotic hands that provide low-precision control, robust in-grasp manipulation is challenging. Abundant tactile information could provide the required feed-back to achieve reliable in-grasp manipulation also in such cases. In this research, soft distributed 3-axis skin sensors ("uSkin") and 6-axis F/T (force/torque) sensors were mounted on each fingertip of an Allegro Hand to provide rich tactile information. These sensors yielded 78 measurements for each fingertip (72 measurements from the uSkin and 6 measurements from the 6-axis F/T sensor). However, such high-dimensional tactile information can be difficult to process because of the complex contact states between the grasped object and the fingertips. Therefore, a convolutional neural network (CNN) was employed to process the tactile information. In this paper, we explored the importance of the different sensors for achieving in-grasp manipulation. Successful in-grasp manipulation with untrained daily objects was achieved when both 3-axis uSkin and 6-axis F/T information was provided and when the information was processed using a CNN.

Performing various in-hand manipulation tasks, without learning each individual task, would enable robots to act more versatile, while reducing the effort for training. However, in general it is difficult to achieve stable in-hand manipulation, because the contact state between the fingertips becomes difficult to model, especially for a robot hand with anthropomorphically shaped fingertips. Rich tactile feedback can aid the robust task execution, but on the other hand it is challenging to process high-dimensional tactile information. In the current paper we use two fingers of the Allegro hand, and each fingertip is anthropomorphically shaped and equipped not only with 6-axis force-torque (F/T) sensors, but also with uSkin tactile sensors, which provide 24 tri-axial measurements per fingertip. A convolutional neural network is used to process the high dimensional uSkin information, and a long short-term memory (LSTM) handles the time-series information. The network is trained to generate two different motions ("twist" and "push"). The desired motion is provided as a task-parameter to the network, with twist defined as -1 and push as +1. When values between -1 and +1 are used as the task parameter, the network is able to generate untrained motions in-between the two trained motions. Thereby, we can achieve multiple untrained manipulations, and can achieve robustness with high-dimensional tactile feedback.

Future robots are expected to share their workspace with humans. Controlling and limiting the forces that such robots exert on their environment is crucial. While force control can be achieved actively with the help of force sensing, passive mechanisms have no time delay in their response to external forces, and would therefore be preferable. Series clutch actuators can be used to achieve high levels of safety and backdrivability. This work presents the first implementation of a linear series clutch actuator. It can exert forces of more than 110N while weighing less than 2kg. Force controllability and safety are demonstrated. Static balancing, which is important for the application in a vertical joint, is also implemented. The power consumption is evaluated, and for a payload of 3kg and with the maximum speed of 94mm/s, the power consumed by the actuator is 11W. Overall, a practical implementation of a linear series clutch actuator is reported, which can be used for future collaborative robots.

In past research, in-hand object manipulation for various sized and shaped objects has been achieved. However, the network had to be trained for each different motion. Training data takes time to acquire and increases the hardware load, thereby increasing the cost for training data. Four-fingered in-hand manipulation is especially difficult as a high number of joints need to be controlled in synchrony. This paper presents a method that reduces the required training data for in-hand manipulation with the idea of pretraining and mutual finger motions. The Allegro Hand is used with soft fingertips and integrated 6-axis F/T sensors to evaluate the proposed method. To make the network more versatile, the training data included objects of various sizes and shapes. When pretraining the network, one shot learning suffices to learn a new task; mutual finger motions can be exploited to use three-fingered pretraining data for four-fingered manipulation. Both data-sharing and weight-sharing were used and show similar results. Crucially, pretraining data from fingers with the same kinematic chain has to be used, showing the importance of morphology specific learning. Moreover, objects with untrained sizes and shapes could be manipulated.

Distributed tactile sensors on multi-fingered hands can provide high-dimensional information for grasping objects, but it is not clear how to optimally process such abundant tactile information. The current paper explores the possibility of using a morphology-specific convolutional neural network (MS-CNN). uSkin tactile sensors are mounted on an Allegro Hand, which provides 720 force measurements (15 patches of uSkin modules with 16 triaxial force sensors each) in addition to 16 joint angle measurements. Consecutive layers in the CNN get input from parts of one finger segment, one finger, and the whole hand. Since the sensors give 3D (x, y, z) vector tactile information, inputs with 3 channels (x, y and z) are used in the first layer, based on the idea of such inputs for RGB images from cameras. Overall, the layers are combined, resulting in the building of a tactile map based on the relative position of the tactile sensors on the hand. Seven different combination variations were evaluated, and an over-95% object recognition rate with 20 objects was achieved, even though only one random time instance from a repeated squeezing motion of an object in an unknown pose within the hand was used as input.

This paper investigates tactile object recognition with relatively densely distributed force vector measurements and evaluates what kind of tactile information is beneficial for object recognition. The uSkin tactile sensors are embedded in an Allegro Hand, and provide 240 triaxial force vector measurements in total in all fingers. Active object sensing is used to gather time-series training and testing data. A simple feedforward, a recurrent, and a convolutional neural network are used for recognizing objects. Evaluations with different number of employed measurements, static vs. time series data and force vector vs. only normal force vector measurements show that the high-dimensional information provided by the sensors is indeed beneficial. An object recognition rate of up to 95% for 20 objects was achieved.

Since the deceleration of the markets of infrastructures in developed economies occurred, the growing demand for infrastructure development in semi-developed countries has become more outstanding. Semi-developed countries such as Thailand aims to become a developed country. However, it is stuck in 'semi-developed country trap'. To break through this, Thai people need local industries with high-added value. Therefore, we proposed an inversed innovative strategy focusing on railway industry. At first, how railway industry was established so far was investigated for finding keys to establish it. Next, we had field trips so that we can find real needs from Thai people. Actually, many countries compete in Thai railway markets. Thus, a comparison between our proposal and others were made. Our proposal for establishing railway industry includes some steps to get local industry focusing on Thai situation. Finally, it also presents ASEAN market as a future plan after they acquire their local industries.

In-hand manipulation is often needed to accomplish a practical task after grasping an object. In-hand manipulation of variously sized and shaped objects in multi-fingered hands without dropping the object is challenging. In this paper we suggest a combined strategy of force control and passive adaptation through soft fingertips with simple interpolation control to achieve in-hand manipulation between various postures and with various objects. While passive compliance can be achieved in numerous ways, this paper uses soft skin, as it does not require complex mechanisms and was easy to integrate in the robot hand (Allegro hand). Softness has proven to significantly ease object grasping, and the current paper shows the importance of softness also for in-hand manipulation. In particular, the simple interpolation strategy between various postures is successful when combined with soft fingertips, with or without force control, but fails with hard fingertips. Objects of varying size, shape and hardness were reliably manipulated. While the soft fingertips enabled good results in our experiments, a sufficiently precise definition of the postures and object size was required. When combining the interpolation control with a force control strategy, bigger errors in defining the posture and object size are possible, without deforming or dropping the object, and the resultant force is lower. As a result, we achieved robust in-hand manipulation between various postures and with objects of different size, shape and hardness.

Moving objects within the hand is challenging, especially if the objects are of various shape and size. In this paper we use machine learning to learn in-hand manipulation of such various sized and shaped objects. The TWENDY-ONE hand is used, which has various properties that makes it well suited for in-hand manipulation: a high number of actuated joints, passive degrees of freedom and soft skin, six-axis force/torque (F/T) sensors in each fingertip, and distributed tactile sensors in the skin. A dataglove is used to gather training samples for teaching the required behavior. The object size information is extracted from the initial grasping posture. After training a neural network, the robot is able to manipulate objects of untrained sizes and shape. The results show the importance of size and tactile information. Compared to interpolation control, the adaptability for the initial posture gap could be greatly extended. Final results show that with deep learning the number of required training sets can be drastically reduced.