Electrical resistance tomography (ERT) can be used to create large-scale soft tactile sensors that are flexible and robust. Good performance requires a fast and accurate mapping from the sensor’s sequential voltage measurements to the distribution of force across its surface. However, particularly with multiple contacts, this task is challenging for both previously developed approaches: physics-based modeling and end-to-end data-driven learning. Some promising results were recently achieved using sim-to-real transfer learning, but estimating multiple contact locations and accurate contact forces remains difficult because simulations tend to be less accurate with a high number of contact locations and/or high force. This paper introduces a modular hybrid method that combines simulation data synthesized from an electromechanical finite element model with real measurements collected from a new ERT-based tactile sensor. We use about 290 000 simulated and 90000 real measurements to train two deep neural networks: the first (Transfer-Net) captures the inevitable gap between simulation and reality, and the second (Recon-Net) reconstructs contact forces from voltage measurements. The number of contacts, contact locations, force magnitudes, and contact diameters are evaluated for a manually collected multi-contact dataset of 150 measurements. Our modular pipeline’s results outperform predictions by both a physics-based model and end-to-end learning. Note to Practitioners–ERT-based tactile sensors use high-speed voltage measurements from electrodes distributed over a piezoresistive area to output a force map that shows where contact is occurring, and how strong each contact is. Such sensors hold promise for giving robots and other devices a sense of touch over large surfaces with low hardware complexity. However, the software problem of converting voltages to an accurate force map has not previously been solved well, requiring either extensive model calibration or extensive contact data collection. This paper suggests a hybrid approach where a straightforward physics model simulates multi-contact experiments that are too costly to acquire in reality and a practical automatic indentation setup acquires real but geometrically limited multi-contact data. Although the number of real measurements required to learn the discrepancy between the sensor and the model is still large due to the inherent inverse nature of ERT-based tactile sensors, our combination of simulation and deep networks achieves better performance than either physical modeling or learning alone. This approach can advance practical large-area tactile sensing for industrial automation systems where multiple contacts occur, such as in manufacturing and assistive robotics. It could also likely be adapted to other nonlinear inverse problems.

Tactile sensors based on electrical resistance tomography (ERT) have shown many advantages for implementing a soft and scalable whole-body robotic skin; however, calibration is challenging because pressure reconstruction is an ill-posed inverse problem. This paper introduces a method for calibrating soft ERT-based tactile sensors using sim-to-real transfer learning with a finite element multiphysics model. The model is composed of three simple models that together map contact pressure distributions to voltage measurements. We optimized the model parameters to reduce the gap between the simulation and reality. As a preliminary study, we discretized the sensing points into a 6 by 6 grid and synthesized single- and two-point contact datasets from the multiphysics model. We obtained another single-point dataset using the real sensor with the same contact location and force used in the simulation. Our new deep neural network architecture uses a de-noising network to capture the simulation-to-real gap and a reconstruction network to estimate contact force from voltage measurements. The proposed approach showed 82% hit rate for localization and 0.51 N of force estimation error performance in singlecontact tests and 78.5% hit rate for localization and 5.0 N of force estimation error in two-point contact tests. We believe this new calibration method has the possibility to improve the sensing performance of ERT-based tactile sensors.

Electrical resistance tomography (ERT) has previously been utilized to develop a large-scale tactile sensor because this approach enables the estimation of the conductivity distribution among the electrodes based on a known physical model. Such a sensor made with a stretchable material can conform to a curved surface. However, this sensor cannot fully cover a cylindrical surface because in such a configuration, the edges of the sensor must meet each other. The electrode configuration becomes irregular in this edge region, which may degrade the sensor performance. In this paper, we introduce an ERT-based robotic skin with evenly and sparsely distributed electrodes. For implementation, we sprayed a carbon nanotube (CNT)-dispersed solution to form a conductive sensing domain on a cylindrical surface. The electrodes were firmly embedded in the surface so that the wires were not exposed to the outside. The sensor output images were estimated using a deep neural network (DNN), which was trained with noisy simulation data. An indentation experiment revealed that the localization error of the sensor was 5.2 ± 3.3 mm, which is remarkable performance with only 30 electrodes. A frame rate of up to 120 Hz could be achieved with a sensing domain area of 90 cm2. The proposed approach simplifies the fabrication of 3D-shaped sensors, allowing them to be easily applied to existing robot arms in a seamless and robust manner.

Robots operating in unstructured environments would benefit from soft whole-body tactile sensors, but implementing such systems typically requires complex electrical wiring to a large number of sensing elements. The reconstruction method called electrical resistance tomography (ERT) has shown promising results (good coverage, manufacturability, and robustness) using electrodes located only along the boundary of the sensing region. However, relatively poor spatial resolution in the sensor's central region is a major drawback of the ERT approach. This paper introduces a new scheme of internal array electrodes to improve spatial resolution. We also systematically derive the optimal pairwise current injection patterns from a mathematical formulation of the ERT system. By highlighting the importance of each electrode pair, this approach enabled us to reduce the number of current injection patterns. Simulation of the standard and proposed sensor designs revealed that the internal array electrodes greatly improve distinguishability in the central region. For validation, a fabric-based soft tactile sensor made of multiple conductive fabrics was developed, including electronics that enable sampling at 200 Hz. During a 225-point localization test conducted without sensor-specific calibration, the constructed sensor showed average localization errors of 2.85 cm ± 1.02 cm. This result is notable because only 16 point electrodes were used to achieve this performance.

Recently, a whole-body tactile sensing have emerged in robotics for safe human-robot interaction. A key issue in the whole-body tactile sensing is ensuring large-area manufacturability and high durability. To fulfill these requirements, a reconstruction method called electrical impedance tomography (EIT) was adopted in large-area tactile sensing. This method maps voltage measurements to conductivity distribution using only a few number of measurement electrodes. A common approach for the mapping is using a linearized model derived from the Maxwell's equation. This linearized model shows fast computation time and moderate robustness against measurement noise but reconstruction accuracy is limited. In this paper, we propose a novel nonlinear EIT algorithm through Deep Neural Network (DNN) approach to improve the reconstruction accuracy of EIT-based tactile sensors. The neural network architecture with rectified linear unit (ReLU) function ensured extremely low computational time (0.002 seconds) and nonlinear network structure which provides superior measurement accuracy. The DNN model was trained with dataset synthesized in simulation environment. To achieve the robustness against measurement noise, the training proceeded with additive Gaussian noise that estimated through actual measurement noise. For real sensor application, the trained DNN model was transferred to a conductive fabric-based soft tactile sensor. For validation, the reconstruction error and noise robustness were mainly compared using conventional linearized model and proposed approach in simulation environment. As a demonstration, the tactile sensor equipped with the trained DNN model is presented for a contact force estimation.

This paper presents a printable skin adhesive stretch sensor to estimate rotation angles of multi-axis joint for biomedical engineering applications, such as gait analysis, gesture recognition, and motion monitoring. Silicone rubber mixed with multiwall carbon nanotube composites were fabricated to make a highly stretchable (up to 120%) strain sensors. Embedding liquid state composites to the fabric leaded to enhance the adhesive force between the composites and fabric. The skin adhesive sensor system could be used to estimate rotation angles of the multi-axis joints while providing comfortable physical interfaces. To reduce misalignment to the axis of rotation, a calibration method was formulated based on geometrical relationship between the sensor axes and the joint angle. In order to validate the estimation performance of the sensor in multi-axis joint, shoulder flexion/extension and abduction/adduction angles were estimated. The results showed the system could estimate the rotation angles of the shoulder enough to identify motion intentions of the user. In the future, the proposed system can be applied to the motion monitoring system by direct attachment to skin without discomfort to the user in daily life.

Medical robotic system is a reliable method to provide a precise and safe treatment to the patients, especially in the area of urology. The prostate cancers are firmer than normal tissues. Therefore, physicians can detect the prostate cancer with a palpation. In this paper, we developed a robotic system to detect prostate cancers using robotic palpation to find the suspected area of the tissue as well as to localize the area of the needle biopsy. To validate the performance of the developed system, the experiment on soft tissue phantoms was carried out. The results indicate that the system can identify the differences of tissue phantoms within and without inclusions. In addition, the bigger inclusions and the closer inclusions from the surface show larger value of the force response. These results indicate that the developed robotic system can be used to find hard materials such as tumor and malignant tissue in the prostate.