Currently, deep neural networks can be extremely effective in robotic tactile perception. However, a major challenge is to solve the problem of continual learning of robotic tactile perception in an open and dynamic environment. In this paper, we propose a novel continual learning method for the domian incremental learning task in the field of tactile perception. To be specific, we introduce a morphology-specific variational autoencoders which can mitigate catastrophic forgetting by generating pseudo-samples for training in the continual learning process. We integrate the generative model and the discriminative model into one model, which reduces the size of model and improves the continual learning ability. In addition, considering the ordinal information between the hardness levels, we propose to add conditional information to the model and introduce a modified loss function to combine the latent value with the hardness information, which improves the continual learning performance by controlling the distribution and quality of pseudo-sample generation. Following this, we designed a tactile robot experiment, collected hardness data, and tested our model on this object hardness recognition task. We show experimentally that, after training, the model can still maintain the accuracy of more than 94% after learning three tasks in terms. Note to Practitioners—In the field of robotics tactile perception, the issue of continual learning in robots is a crucial problem that urgently requires resolution. We hope robots to effectively engage in continual learning across multiple tasks, ensuring the acquisition of new knowledge while mitigating the risk of forgetting previously acquired knowledge. In this paper, we propose a novel continual learning method for the domian incremental learning task. we introduce a morphology-specific variational autoencoders based on replaying pseudo-samples during continual learning process which reduces the size of model and improves the continual learning ability. We enhance model performance by integrating generative and discriminative models, incorporating conditional information to control the distribution of replayed sample types, and leveraging sequential relationships among samples. It is proved that the proposed method is able to effectively improve the accuracy in a tactile domian incremental learning task.

This article addresses the robustness of unmanned surface vehicles (USV) using model-based reinforcement learning (MBRL). A novel MBRL approach, Lyapunov probabilistic model predictive control (LPMPC) is proposed to simultaneously learn both the probabilistic model of a USV and its corresponding estimated Lyapunov region of attraction (ROA) under one reinforcement learning framework. Unlike the existing MBRL USV systems with less consideration of robustness and safety, our method naturally learns a general indicator of system stability based on the probabilistic model’s belief and employs it to guide its policy. Evaluated by different navigation tasks in a simulation driven by real boat data, LPMPC demonstrated significant advantages in both control robustness and task completion against various levels of environmental disturbances compared with the baseline approach without Lyapunov ROA’s guidance. Note to Practitioners—Modelling the system stability without human prior knowledge is challenging in the domain of USV. This work proposed a data-driven method to iteratively learn a task-relevant stability model of USV in a probabilistic view. Based on the evaluation of a real boat data-driven simulation, the learned stability model contributed to superior driving skills in different USV scenarios by properly indicating and avoiding potentially risky states. In future research, we plan to expand the definition of risks in different tasks, such as loss of control, overlarge sway, and excessive energy consumption and investigate the proposed approach in real-world USV.

Inadequate consideration of tactile sensor features in existing algorithms can lead to insufficient prediction accuracy for robotic tactile recognition tasks. In this paper, we propose a spatial-temporal information based adjacency matrix construction method called transfer cost. On this basis, we propose a multi-branch multi-scale channel fusion graph convolutional network (MSCF-LSTM-GCN). This network combines the advantages of long short-term memory networks (LSTM) with graph convolutional networks (GCN) to extract spatial-temporal features. It is the first LSTM-GCN network with a multi-branch structure that utilizes multi-feature scale and multi-channel fusion. This structure provides diverse perceptual ranges and information levels. We introduce shortcuts combined with the multi-branch structure to reduce the loss of information caused by multiple layers of transmission. The Einstein summation calculation method and the parameters of binarizing the adjacency matrix are optimized to enable the established dynamic tactile graph to participate in batch training. The proposed method improves both prediction accuracy and F1 score by over 1% on the tactile grasp stability prediction task. In the tactile object recognition task, the proposed method improves both prediction accuracy and F1 score by over 3%.Note to Practitioners—Robotic tactile perception is an important research area that can improve the flexibility, accuracy and stability of robots in robotic tactile recognition tasks. However, the amount of accessible information depends on the resolution and number of tactile sensors, and the features extracted using conventional deep learning methods are limited. For this reason, we propose a spatial-temporal information based adjacency matrix construction method called transfer cost. Then, we propose the MSCF-LSTM-GCN model that extract spatial-temporal features from different feature scales and shortcuts. The network solves the problems of traditional deep learning methods in acquiringsingle features and losing deep information. The experiments on two public datasets show that our method performs due to other competing algorithms on the tasks of robotic grasp stability prediction and object recognition for multi-tactile sensor scenarios. In future research we will invest in the problem of 3D force regression.

Tiny crack detection plays a vital role in ensuring the safety of critical industrial components and infrastructure, providing early intervention to prevent crack propagation and mitigate potential damage. Although machine vision-based defect detection has been greatly utilized in safety inspection and maintenance, tiny crack detection remains a big issue due to the low visibility and weak features with large background noise. To address this challenge, this paper presents an end-to-end trainable hierarchical constrained neural network for tiny crack detection. Firstly, we present a hierarchical loss constraint Filter (HLCF) module based on a region-level attention mechanism to capture the holistic vein structure of cracks and enhance the faint feature extraction of tiny cracks. In order to balance local high contrast and holistic crack features, a feature fusion loss constraint is designed to reduce noise interference and refine the boundary details of cracks by learning different receptive fields in each layer. Besides that, we created a dataset consisting of tiny crack samples collected through image processing and manual labeling from industrial production, named TinyCrack. The proposed HLCF model is evaluated on TinyCrack and seven public databases. The experimental indices show that the model achieves Precision over 0.62%, Recall over 0.12%, F-score over 0.53%, and IOU over 1.12% in TinyCrack, demonstrating the best accuracy compared with other public crack detection models. The results indicate that the HLCF detection model has better performance in identifying low-visibility tiny cracks.

The sense of touch is essential for robots to perform various daily tasks. Artificial Neural Networks have shown significant promise in advancing robotic tactile learning. However, due to the changing of tactile data distribution as robots encounter new tasks, ANN-based robotic tactile learning suffers from catastrophic forgetting. To solve this problem, we introduce a novel continual learning (CL) framework called the Probabilistic Spiking Neural Network with Variational Continual Learning (PSNN-VCL). In this framework, PSNN introduces uncertainty during spike emission and can apply fast Variational Inference by optimizing the uncertainty through backpropagation, which significantly reduces the required model parameters for VCL. We establish a robotic tactile CL benchmark using publicly available datasets to evaluate our method. Experimental results demonstrated that, compared to other CL methods, PSNN-VCL not only achieves superior performance in terms of widely used CL metrics but also achieves at least a 50% reduction in model parameters on the robotic tactile CL benchmark.

Multi-sensor based motion tracking is of great interest to the robotics community as it may lessen the need for expensive optical motion capture equipment. However, the traditional convolution algorithms have difficulty adapting to the data due to the changes of joints’ relative position during motion. The time-series networks often used in the past ignore the spatial characteristics of sensors. We tackle this challenge by combining long short-term memory (LSTM) with graph convolution network (GCN), adding the prior knowledge of sensor distribution, and integrating it into the motion law through the adjacency matrix. This article proposes a novel shortcut enhanced LSTM-GCN network (SE-LSTM-GCN). It connects LSTM and GCN in sequence and extracts temporal and spatial features of data. At the same time, the shortcut is used in the network to enhance the output of two middle layers and to restore the filtered information. Our experimental results on two different motion tracking datasets show that the proposed network is able to learn the mapping relationship with better universality, less tracking error, and without increasing much training time, and can better perform human motion tracking tasks.Note to Practitioners—Accurate and real-time multiple soft sensors motion tracking suits are more accepted for their low cost. However, the soft-sensor based motion tracking is not comparable to the traditional optical equipment in prediction error. To this end, we present a novel network shortcut enhanced LSTM-GCN (SE-LSTM-GCN), consisting of shortcuts, long short-term memory (LSTM), and graph convolution network (GCN). The LSTM solves the non-linear and hysteresis of soft strain sensors, and GCN is integrated into the network since the knowledge of sensor location can be put into the adjacency matrix generated by the k-nearest neighbor (KNN). While shortcuts are used to enhance the output of middle layers to form combined features. Experimental results on two public datasets show that the proposed network is superior to competing algorithms in terms of prediction error. The network can be deployed in embedded devices, such as VR gloves to provide a better gaming experience. The current algorithm is based on the relationship between sensor data and distance. In future research, we will focus on adding other human kinematics laws to the network.

Accurately estimating uncertain body disturbances is critical for the effective integration of wearable exoskeletons for active human users. In this article, considering nonlinear time-varying human disturbances, we propose a TDE-BVC feedback control method that performs biomimetic viscoelastic compliance (BVC) with transformer-based deformation estimator (TDE). The method provides human-exoskeleton with stable and dynamic walking capabilities. We developed a transformer-based end-to-end deformation estimation sequence network that simultaneously captures the mapping relationship between foot force/torque and exoskeleton deformation. Moreover, we integrated the BVC to eliminate the impact and external disturbances experienced by the human-exoskeleton, enabling it to closely follow a reference gait, and utilized Lyapunov’s theorem to prove its stability. The control strategy is independent of the parameters of the human exoskeleton. To evaluate the effectiveness of the proposed method, walking experiments were conducted on different subjects. Our results indicate that with only 6-axis force/torque sensors, the TDE-BVC controller could accurately estimate and compensate for the deformation of different human-exoskeletons in each control cycle$(p\lt 0.001)$, with robustly stable and adaptive dynamic walking within a bounded control error.

Hardness is one of the most critical tactile properties for robots to recognize objects. Machine learning methods have shown superior performance in object hardness classification. However, existing machine learning methods for tactile hardness classification cannot use the ordinal information between hardness classes because the one-hot encoding only cares about the correct class and ignores the inter-class relationship. To solve this problem, we propose to generalize the one-hot encoding using unimodal distributions including the Poisson and binomial distributions for tactile ordinal classification problems, resulting in two tactile ordinal networks (TacONet): TacONet-p and TacONet-b. Furthermore, we collect a tactile hardness dataset on the silicone samples with three different shapes (Shapes A, B, C), and each shape samples have thirteen hardness classes ranging from 0A (Shore A scale) to 60A at 5A intervals. We validate the resulting method for tactile hardness classification using a real robot. Experimental results demonstrate that compared with state-of-the-art methods, the proposed method achieves better classification performance in terms of accuracy and quadratic weighted kappa (QWK) on the tactile hardness dataset, reaching a classification accuracy up to 99.5% and a QWK up to 99.9% on Shape C. Note to Practitioners—In the field of robotics tactile recognition, hardness classification is one of the most important and common tasks for robots to accurately recognize objects, particularly when the environment is dark or visual sensors are not working. In this paper, we propose a novel tactile ordinal network for tactile hardness classification tasks. The existing machine learning models for tactile hardness classification are trained by minimizing the cross-entropy loss between predicted vectors and one-hot encoding vectors of true classes, which makes the models only care about the correct classes and ignores the inter-class relationship of hardness classes. In other words, these models have the same probability to misclassify a hardness class with any other hardness class. To tackle this problem, we propose to generalize the one-hot encoding method using a unimodal distribution method to encode the true classes. The unimodal distribution encoding vectors can make the model learn the ordinal information between classes. It is proved that the proposed method is able to effectively improve the classification accuracy and QWK in a tactile hardness classification task.

Multisource biosignals are being increasingly used in human–machine interaction applications. In particular, a critical problem in exoskeleton-assisted rehabilitation for patients with hemiplegia is the generation of rhythmic and symmetrical locomotion. To support lower limb rehabilitation, we propose a model for predicting contralateral joint angles using multisource biosignals. First, a vibroarthrography (VAG) sensor is attached to the affected leg, and surface electromyography sensors are attached to the sound leg. The corresponding signals are used to estimate the hip, knee, and ankle joint angles of the affected leg. Second, an algorithm based on a temporal convolution network (TCN) is introduced to predict the contralateral lower-limb joint angles during human locomotion. The TCN is compared with a long short-term memory (LSTM) network and a convolutional neural network. Experiments were conducted by 10 healthy participants. The results of the proposed model were compared with measurements from encoders in three joints mounted on an exoskeleton to verify the applicability of the proposed model. In addition, by using the outputs of a motion capture system as the ground truth, the experimental results validated the model prediction performance. The prediction root mean square error (RMSE) of the TCN was 52%–70% lower than that of the LSTM network and CNN at different paces. The prediction RMSE using VAG was 20%–24% lower than that without using VAG. Note to Practitioners—We propose a model for predicting the contralateral lower-limb joint angles of an exoskeleton. The model uses measurements from multiple biosignal sensors to support exoskeleton-assisted hemiplegia rehabilitation. A sensing system based on acoustic signals and bioelectric signals is developed to improve the joint angle prediction in the affected leg of patients with hemiplegia. A temporal convolution network is implemented to estimate joint angles, and training considering the root mean square error and Pearson correlation coefficient as evaluation indicators is performed. Unlike similar methods that include no sensory feedback from the affected leg, the proposed method incorporates VAG signals from the affected leg and sEMG signals from the sound leg to handle different degrees of hemiplegia and support the generation of rhythmic and symmetrical locomotion for gait rehabilitation using an exoskeleton.

Aiming at the problem that it is difficult to calculate the force of permanent magnets in the magnetic field, this paper proposes a nonlinear mechanical model of linear array magnetic field based on radial basis function neural network (RBFNN). Combined with the linear Halbach array adsorption module of the wall-climbing robot, the three-dimensional geometric magnetic fields of four typical linear array permanent magnets were constructed, and the theoretical models of the interaction between the magnetic fields were given respectively. Further, the finite element simulation calculation of the magnetic force was carried out using COMSOL Multiphysics. According to the parametric scanning results of the orthogonal test, a nonlinear intelligent prediction model of the force between magnetic fields with local loss sensitivity is established by using the RBFNN numerical fitting method. The average deviation of the network test set is 1.19, and the standard deviation is 0.80. The intelligent prediction model has strong generalization performance, faster convergence speed and stronger flexibility, which provides a theoretical basis for the interaction and control of array magnetic fields.

Touch modality identification has attracted increasing attention due to its importance in human–robot interactions. There are three issues involved in the tactile perception for the touch modality identification, including the high dimensionality of tactile signals, complex tensor morphology of tactile sensing units, and the misalignment among different tactile time-series samples. In this article, we propose a novel kernel-based approach to deal with these three issues in a unified framework. Specifically, the techniques, including sparse principal component analysis and subsampling, are employed to reduce the feature dimension. Then, a singular value decomposition (SVD)-based kernel is proposed to preserve the spatial information of the tactile sensing elements. The sample misalignment issue is addressed via the employment of a global alignment kernel. Moreover, the merits of these two kernels are fused through an ideal regularized composite kernel, which simultaneously takes the label information of the training set into consideration. The effectiveness of the proposed kernel-based approach is verified on a public touch modality data set with a comprehensive comparison with the competing methods.Note to Practitioners—In a wealth of tactile recognition tasks, we are in the face of various challenges. For instance, tactile measurements are commonly tensorial and high-dimensional. The misalignments among tactile measurements prevail, such as different durations of tactile measurements and the misaligned starting time point of tactile measurements. This article presents a kernel-based method using an ideal regularized composite kernel to deal with all challenges in a unified framework. The kernel-based method consists of two key components including the SVD-based kernel and the global alignment kernel. The proposed method may shed new insights on new advances in tactile signal processing particularly in human–robot interactions.

Sensor drift is one of the severe issues that gas sensors suffer from. To alleviate the sensor drift problem, a gas sensor drift compensation approach is proposed based on local discriminant subspace projection (LDSP). The proposed approach aims to find a subspace to reduce the distribution difference between two domains, i.e., the source and target domain. Similar to domain regularized component analysis (DRCA) which is a recently proposed sensor drift correction method, the mean distribution discrepancy is minimized in the common subspace in our approach. LDSP extends DRCA in two aspects, i.e., it not only takes the label information of the source data into consideration to reduce the possibility of the case that samples in the subspace with different class labels stay close to each other, but also borrows the idea of locality-preserving projection to deal with multimodal data. Specifically, inspired by local Fisher discriminant analysis (LFDA), the label information is utilized to maximize the local between-class variance of source data in the latent common subspace and simultaneously minimize the local within-class variance. The formulation of LDSP is a generalized eigenvalue problem that can be readily solved. The experimental results have shown the proposed method outperforms other gas sensor drift compensation methods in terms of classification accuracy on two public gas sensor drift datasets.

This article presents a novel discriminative subspace-learning-based unsupervised domain adaptation (DA) method for the gas sensor drift problem. Many existing subspace learning approaches assume that the gas sensor data follow a certain distribution such as Gaussian, which often does not exist in real-world applications. In this article, we address this issue by proposing a novel discriminative subspace learning method for DA with neighborhood preserving (DANP). We introduce two novel terms, including the intraclass graph term and the interclass graph term, to embed the graphs into DA. Besides, most existing methods ignore the influence of the subspace learning on the classifier design. To tackle this issue, we present a novel classifier design method (DANP+) that incorporates the DA ability of the subspace into the learning of the classifier. The weighting function is introduced to assign different weights to different dimensions of the subspace. We have verified the effectiveness of the proposed methods by conducting experiments on two public gas sensor datasets in comparison with the state-of-the-art DA methods.

Tactile sensing with spiking neural networks (SNNs) has attracted increasing attention in the past decades. In this article, a novel SNN framework is proposed for the tactile surface roughness categorization task. In contrast to supervised SNN methods such as ReSuMe and Tempotron that require prespecifying target spike trains, the presented method performs the classification through directly comparing the distance between multineuron spike trains. Unlike simple spike train fusion methods using average pairwise spike train distance or pooled spike train distance, the proposed method merges spike trains from different neurons with the multineuron spike train distance, which can capture the complex correlation of multiple spike trains. Specifically, the spike trains are generated via the Izhikevich neurons from tactile signals. The similarity of the multineuron spike trains is computed using the multineuron Victor–Purpura spike train distance, which can be efficiently implemented in an inductive manner. The classification can be performed by incorporating$k$-nearest neighbors and the multineuron spike train distance as a similarity metric. The proposed framework is quite general, i.e., other multineuron spike train distances and spike train kernel-based methods can be readily incorporated. The effectiveness of the proposed method has been demonstrated on a tactile data set by comparing it with various feature- and spike-based methods.Note to Practitioners—In the soft neuromorphic implementation of biomimetic tactile sensing and the development of the tactile sensing capability in neurobotic systems, the processing and analysis of spike-like tactile signals are quite common. Inspired by human tactile perception, this article proposes a novel supervised spiking neural network method for tactile sensing tasks. The traditional methods have to prespecify target spike trains, which is still an open question. In addition, the current ways to fuse spike trains from multiple neurons are far from mature. This article tackles these two problems using spike train similarity comparison with multineuron spike train distance. The direct spike train similarity comparison avoids the need to prespecify target spike trains. The multineuron spike train distance can inherently fuse spike trains from different neurons. It is demonstrated that the proposed method is able to effectively perform classification in a tactile roughness discrimination task.

In this paper, we propose a novel method for multimodal material classification based on ensemble learning and optimized features. The proposed method consists of three key steps. Firstly, we extract a set of features for each modality. Compared to existing methods, the extracted features are relatively simple but more effective when they are incorporated into the classifiers. Then, the feature selection algorithms, including Multi-Cluster Feature Selection (MCFS) and Laplacian Score (LS) are employed to reduce the feature dimension due to the curse of dimensionality. Finally, an ensemble learning method is proposed to integrate the merits of different feature selection methods. The effectiveness of the proposed method is demonstrated on the LMT-108 surface material dataset which includes multiple modalities such as sound, acceleration, and image. The experimental results have shown that our approach performs better than the competing methods.

Accurate object shape knowledge provides important information for performing stable grasping and dexterous manipulation. When modeling an object using tactile sensors, touching the object surface at a fixed grid of points can be sample inefficient. In this paper, we present an active touch strategy to efficiently reduce the surface geometry uncertainty by leveraging a probabilistic representation of object surface. In particular, we model the object surface using a Gaussian process and use the associated uncertainty information to efficiently determine the next point to explore. We validate the resulting method for tactile object surface modeling using a real robot to reconstruct multiple, complex object surfaces.