Dexterous hands are the key element for robots to achieve human-like manipulation capabilities. An outstanding challenge is to provide fingertips of dexterous hands with precise tactile deformation sensing capabilities. In this paper, we present the GelStereo Tip, a spherical and easy-to-integrate GelStereo-type visuotactile sensor capable of sensing high-resolution 3D elastomer deformation. Previous calibration method does not take into account the impact of imaging errors caused by the sensor’s compact and high-curvature structural characteristics on the accuracy of tactile sensing. Therefore, we propose a novel self-calibration method based on the Refractive Stereo Ray Tracing model, named GTSC, and demonstrate the accuracy of less than 0.3 mm for deformation sensing. Furthermore, we also propose a Contact Retention Tactile Controller to address the issue of fingertips being unable to overcome obstructive torque during the multi-finger bottle cap screwing. After integrating GelStereo Tip into fingertips of Allegro Hand, the controller adjusts the joint positions of the given trajectory using proportional control based on the difference between the sensor’s actual deformation and the reference state for contact retention. We believe that the GelStereo Tip sensor combined with robotic dexterous hands has great application potential in the field of multi-finger fingertip manipulation. Note to Practitioners—The motivation of this paper is to design a fingertip visuotactile sensor with high-precision 3D tactile deformation sensing capabilities for multi-finger robotic hands and to validate its sensing performance. Additionally, it aims to address the issue of overcoming resistance in multi-finger screwing manipulations. Currently, most sensors do not consider the refraction effect or ignore the impact of planar imaging errors in refractive calibration. This paper proposes a visuotactile sensor along with a corresponding self-calibration method to ensure its sensing accuracy. Experiments show that our sensor possesses high-precision and robust 3D deformation sensing capabilities. On the other hand, multi-finger hands often struggle to complete screwing tasks along the given trajectory due to disturbances from torque resistance. This paper proposes a tactile controller that evaluates the contact state through aforementioned tactile sensing to improve subsequent trajectory and achieve continuous screwing. Comparative experiments highlight the necessity of this controller and the reliability of tactile sensing. We hope that the design of our sensor, the self-calibration method, and the tactile controller applied to multi-finger screwing can provide new insights for other practitioners.

Visuotactile sensors have been shown to provide rich contact information for robots. However, how to build a high-fidelity visuotactile simulator that supports multi-mode tactile imprints and various sensor configurations (such as coating patterns) remains a challenging problem. In this paper, we present TacFlex, an efficient and flexible simulator for visuotactile sensors, which physically simulates the elastomer deformation using Finite Element Methods (FEM), and focuses on linking the deformed elastomer mesh to diverse tactile imprints, including tactile images with arbitrary coating patterns and tactile 3D point clouds. We further propose a ray tracing-based rectification method to deal with multi-medium refraction effects to make the simulated tactile images more realistic. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness of TacFlex on several visuotactile sensors. Furthermore, we explore the Sim2Real performance of different tactile imprints provided by TacFlex in tactile perception and manipulation tasks, such as cylindrical object pose estimation and peg-in-hole. The perception/policy models trained in simulation are successfully deployed in the real world. Finally, we present the outlook on the potential of TacFlex in visuotactile manipulation learning. The TacFlex simulator is open-sourced to the community. See supplementary video, code, and results athttps://sites.google.com/view/tacflex/.

Abstract Over the past 70 years, the semiconductor industry has undergone transformative changes, largely driven by the miniaturization of devices and the integration of innovative structures and materials. Two-dimensional (2D) materials like transition metal dichalcogenides (TMDs) and graphene are pivotal in overcoming the limitations of silicon-based technologies, offering innovative approaches in transistor design and functionality, enabling atomic-thin channel transistors and monolithic 3D integration. We review the important progress in the application of 2D materials in future information technology, focusing in particular on microelectronics and optoelectronics. We comprehensively summarize the key advancements across material production, characterization metrology, electronic devices, optoelectronic devices, and heterogeneous integration on silicon. A strategic roadmap and key challenges for the transition of 2D materials from basic research to industrial development are outlined. To facilitate such a transition, key technologies and tools dedicated to 2D materials must be developed to meet industrial standards, and the employment of AI in material growth, characterizations, and circuit design will be essential. It is time for academia to actively engage with industry to drive the next 10 years of 2D material research.

In recent years, it is a difficult issue to integrate the deep cross-fertilization and interpretable cognitive modeling methods from the basic theory of emotional psychology with deep learning and other algorithms. To address this problem, a cognitive model that integrates the VGG-facial action coding system (FACS)-OCC model based on fer2013 expression features and the OCC-pleasure-arousal-dominance (PAD)-openness, conscientiousness, extraversion, agreeableness, and neuroticism (OCEAN) fusion of the basic theory of emotional psychology, namely, a computational affection-based OCC-PAD-OCEAN federation cognitive modeling (OPO-FCM), is constructed. By constructing this model and performing formal proof algorithms, it is shown that the OPO-FCM can acquire expression features in video streams, complete the acquisition of expression features in videos by training a deep neural network, map expressions to the PAD emotion space through the established expression–basic emotions–emotion space mapping relationship, and finally complete the mapping of the average emotion over a period time. The information of personality space is obtained through it. Finally, the experimental simulation of the model is conducted, and the results show that the average accuracy of the valid tested personalities is 79.56%. This article takes the knowledge-driven approach of emotional psychology as a starting point and combines deep learning techniques to construct interpretable cognitive models, thus providing new ideas for future cross-innovation between computer technology and psychology theory.

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this article, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3-D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a refractive stereo ray tracing model for GelStereo Palm is presented. Moreover, a 3-D tactile point cloud sensing pipeline is introduced to reconstruct the 3-D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3-D contact geometry sensing of our GelStereo Palm sensor.

Visuotactile sensors have recently attracted much attention in robot communities due to the benefit of high spatial resolution sensing. However, force/torque estimation by visuotactile sensors remains a challenging problem. In this paper, we propose a learning-based six-axis force/torque estimation network using GelStereo visuotactile sensor, which can provide two-dimensional (2D) and three-dimensional (3D) displacements of markers embedded in the sensor surface. The convolutional neural networks are employed to extract multi-modal tactile deformation features; and a novel contact positional encoding method is proposed to eliminate the influence of translation invariance in convolutional operators. The well-trained model achieves the best RMSE of 0.290 N in force and 0.0084 Nm in torque. Furthermore, the proposed force/torque estimation network is integrated with a force-feedback policy for adaptive grasping tasks. The experimental results demonstrate the effectiveness of the proposed method and its potential application in robotic grasping and manipulation tasks.

Unknown surface material classification (SMC) can inform a robot about material properties, enabling it to interact with environments appropriately. Recent research has leveraged multimodal data using deep learning to improve the performance of SMC. In this article, we present a deep learning model, multimodal temporal convolutional neural network (MTCNN), which integrates energy spectrum, dilated convolutions, and sequence poolings into a unified network architecture. The proposed model can learn material representations from auditory and multitactile (i.e., acceleration, normal force, and friction force) data generated by dragging a tool along surfaces, and distinguish unknown object surface materials into categories. For surface material data collection, a tool is also designed to detect different object surfaces. The performance of MTCNN is evaluated on a public dataset and the highest classification accuracy is 87.55%. A robotic curling example is provided to illustrate how the presented model helps the robot in manipulation.

Self-similarity analysis-based feature summarizing technique (SuCo) was proposed recently to improve the time and memory efficiency of Cover Song Identification (CSI). In this paper, both the feature summarizing and the cross-similarity calculating strategies of the SuCo model are modified as follows to enhance its identification accuracy. At the feature summarizing stage, first, the Hubness Reduction (HR) strategy is adopted to reduce the possible `Hubness' phenomenon existing in the feature subsequence community, which may affect the retrieval effectiveness. Then, the Network Enhancement (NE) technique, which was originally proposed in biology to improve gene-function prediction accuracy, is introduced to reduce the noise in the self-similarity network caused by the limitation of feature extraction and similarity measuring, and the inherent musical and acoustic variations. At the cross-similarity calculating stage, first, the summarized representative feature subsequences of the reference are concatenated to obtain its combined representative feature. Then, considering that the nonlinear recurrence property is important for describing the melody perception-based similarity, Qmax is adopted to measure the similarity between the combined representative feature of the reference and the unsummarized feature sequence of the query. Extensive experiments carried out on four open CSI datasets with 5 types of features and 2 kinds of representative feature subsequence choosing methods verify that: i) The proposed scheme outperforms the SuCo model in retrieval effectiveness. ii) Each of the above modifications contributes to the performance enhancement of the proposed scheme. iii) The proposed scheme achieves high generalization.

Opportunistic Network Coding (ONC) has attracted much attention recently. The main improvement of ONC over the traditional network coding is that ONC allows the encoding node to decide whether it employs network coding based on the status of all its input streams. However, due to the nature of fading, wireless links may not always be reliable. Thus, the performance gain of ONC over traditional methods may be diminished due to wireless links in deep fading. Fortunately, some techniques, such as ARQ (Automatic Repeat reQuest) and cooperative diversity etc. can be used to mitigate it. In this paper, we consider the wireless butterfly network topology, a basic component of complex wireless networks. In order to improve the network performance via taking the advantage of ONC, ARQ and cooperative diversity, we propose a new Cooperationbased Opportunistic Network Coding (CP-ONC) protocol and then analyze its performance in terms of network throughput and delay. The advantage of CP-ONC over ONC is that it employs truncated ARQ and cooperative diversity to enhance the reliability of the wireless links. Various simulations show that CP-ONC achieves better gain over ONC in terms of network throughput and delay, especially in low Signal to noise ratio (SNR) region.