Currently, deep neural networks can be extremely effective in robotic tactile perception. However, a major challenge is to solve the problem of continual learning of robotic tactile perception in an open and dynamic environment. In this paper, we propose a novel continual learning method for the domian incremental learning task in the field of tactile perception. To be specific, we introduce a morphology-specific variational autoencoders which can mitigate catastrophic forgetting by generating pseudo-samples for training in the continual learning process. We integrate the generative model and the discriminative model into one model, which reduces the size of model and improves the continual learning ability. In addition, considering the ordinal information between the hardness levels, we propose to add conditional information to the model and introduce a modified loss function to combine the latent value with the hardness information, which improves the continual learning performance by controlling the distribution and quality of pseudo-sample generation. Following this, we designed a tactile robot experiment, collected hardness data, and tested our model on this object hardness recognition task. We show experimentally that, after training, the model can still maintain the accuracy of more than 94% after learning three tasks in terms. Note to Practitioners—In the field of robotics tactile perception, the issue of continual learning in robots is a crucial problem that urgently requires resolution. We hope robots to effectively engage in continual learning across multiple tasks, ensuring the acquisition of new knowledge while mitigating the risk of forgetting previously acquired knowledge. In this paper, we propose a novel continual learning method for the domian incremental learning task. we introduce a morphology-specific variational autoencoders based on replaying pseudo-samples during continual learning process which reduces the size of model and improves the continual learning ability. We enhance model performance by integrating generative and discriminative models, incorporating conditional information to control the distribution of replayed sample types, and leveraging sequential relationships among samples. It is proved that the proposed method is able to effectively improve the accuracy in a tactile domian incremental learning task.

Inadequate consideration of tactile sensor features in existing algorithms can lead to insufficient prediction accuracy for robotic tactile recognition tasks. In this paper, we propose a spatial-temporal information based adjacency matrix construction method called transfer cost. On this basis, we propose a multi-branch multi-scale channel fusion graph convolutional network (MSCF-LSTM-GCN). This network combines the advantages of long short-term memory networks (LSTM) with graph convolutional networks (GCN) to extract spatial-temporal features. It is the first LSTM-GCN network with a multi-branch structure that utilizes multi-feature scale and multi-channel fusion. This structure provides diverse perceptual ranges and information levels. We introduce shortcuts combined with the multi-branch structure to reduce the loss of information caused by multiple layers of transmission. The Einstein summation calculation method and the parameters of binarizing the adjacency matrix are optimized to enable the established dynamic tactile graph to participate in batch training. The proposed method improves both prediction accuracy and F1 score by over 1% on the tactile grasp stability prediction task. In the tactile object recognition task, the proposed method improves both prediction accuracy and F1 score by over 3%.Note to Practitioners—Robotic tactile perception is an important research area that can improve the flexibility, accuracy and stability of robots in robotic tactile recognition tasks. However, the amount of accessible information depends on the resolution and number of tactile sensors, and the features extracted using conventional deep learning methods are limited. For this reason, we propose a spatial-temporal information based adjacency matrix construction method called transfer cost. Then, we propose the MSCF-LSTM-GCN model that extract spatial-temporal features from different feature scales and shortcuts. The network solves the problems of traditional deep learning methods in acquiringsingle features and losing deep information. The experiments on two public datasets show that our method performs due to other competing algorithms on the tasks of robotic grasp stability prediction and object recognition for multi-tactile sensor scenarios. In future research we will invest in the problem of 3D force regression.

The sense of touch is essential for robots to perform various daily tasks. Artificial Neural Networks have shown significant promise in advancing robotic tactile learning. However, due to the changing of tactile data distribution as robots encounter new tasks, ANN-based robotic tactile learning suffers from catastrophic forgetting. To solve this problem, we introduce a novel continual learning (CL) framework called the Probabilistic Spiking Neural Network with Variational Continual Learning (PSNN-VCL). In this framework, PSNN introduces uncertainty during spike emission and can apply fast Variational Inference by optimizing the uncertainty through backpropagation, which significantly reduces the required model parameters for VCL. We establish a robotic tactile CL benchmark using publicly available datasets to evaluate our method. Experimental results demonstrated that, compared to other CL methods, PSNN-VCL not only achieves superior performance in terms of widely used CL metrics but also achieves at least a 50% reduction in model parameters on the robotic tactile CL benchmark.

Hardness is one of the most critical tactile properties for robots to recognize objects. Machine learning methods have shown superior performance in object hardness classification. However, existing machine learning methods for tactile hardness classification cannot use the ordinal information between hardness classes because the one-hot encoding only cares about the correct class and ignores the inter-class relationship. To solve this problem, we propose to generalize the one-hot encoding using unimodal distributions including the Poisson and binomial distributions for tactile ordinal classification problems, resulting in two tactile ordinal networks (TacONet): TacONet-p and TacONet-b. Furthermore, we collect a tactile hardness dataset on the silicone samples with three different shapes (Shapes A, B, C), and each shape samples have thirteen hardness classes ranging from 0A (Shore A scale) to 60A at 5A intervals. We validate the resulting method for tactile hardness classification using a real robot. Experimental results demonstrate that compared with state-of-the-art methods, the proposed method achieves better classification performance in terms of accuracy and quadratic weighted kappa (QWK) on the tactile hardness dataset, reaching a classification accuracy up to 99.5% and a QWK up to 99.9% on Shape C. Note to Practitioners—In the field of robotics tactile recognition, hardness classification is one of the most important and common tasks for robots to accurately recognize objects, particularly when the environment is dark or visual sensors are not working. In this paper, we propose a novel tactile ordinal network for tactile hardness classification tasks. The existing machine learning models for tactile hardness classification are trained by minimizing the cross-entropy loss between predicted vectors and one-hot encoding vectors of true classes, which makes the models only care about the correct classes and ignores the inter-class relationship of hardness classes. In other words, these models have the same probability to misclassify a hardness class with any other hardness class. To tackle this problem, we propose to generalize the one-hot encoding method using a unimodal distribution method to encode the true classes. The unimodal distribution encoding vectors can make the model learn the ordinal information between classes. It is proved that the proposed method is able to effectively improve the classification accuracy and QWK in a tactile hardness classification task.

In this paper, we propose a novel method for multimodal material classification based on ensemble learning and optimized features. The proposed method consists of three key steps. Firstly, we extract a set of features for each modality. Compared to existing methods, the extracted features are relatively simple but more effective when they are incorporated into the classifiers. Then, the feature selection algorithms, including Multi-Cluster Feature Selection (MCFS) and Laplacian Score (LS) are employed to reduce the feature dimension due to the curse of dimensionality. Finally, an ensemble learning method is proposed to integrate the merits of different feature selection methods. The effectiveness of the proposed method is demonstrated on the LMT-108 surface material dataset which includes multiple modalities such as sound, acceleration, and image. The experimental results have shown that our approach performs better than the competing methods.