Abstract Within the field of human–computer interaction, data gloves play an essential role in establishing a connection between virtual and physical environments for the realization of digital human. To enhance the credibility of human‐virtual hand interactions, we aim to develop a system incorporating a data glove‐embedded technology. Our proposed system collects a wide range of information (temperature, bending, and pressure of fingers) that arise during natural interactions and afterwards reproduce them within the virtual environment. Furthermore, we implement a novel traversal polling technique to facilitate the streamlined aggregation of multi‐channel sensors. This mitigates the hardware complexity of the embedded system. The experimental results indicate that the data glove demonstrates a high degree of precision in acquiring real‐time hand interaction information, as well as effectively displaying hand posture in real‐time using Unity3D. The data glove's lightweight and compact design facilitates its versatile utilization in virtual reality interactions.

Merkel cells combine with Aβ afferents, producing slowly adapting type 1(SA1) responses to mechanical stimuli. However, how Merkel cells transduce mechanical stimuli into neural signals to Aβ afferents is still unclear. Here we develop a biophysical model of Merkel cells for mechanical transduction by incorporating main ingredients such as Ca2+ and K+ voltage-gated channels, Piezo2 channels, internal Ca2+ stores, neurotransmitters release, and cell deformation. We first validate our model with several experiments. Then we reveal that Ca2+ and K+ channels on the plasma membrane shape the depolarization of membrane potentials, further regulating the Ca2+ transients in the cells. We also show that Ca2+ channels on the plasma membrane mainly inspire the Ca2+ transients, while internal Ca2+ stores mainly maintain the Ca2+ transients. Moreover, we show that though Piezo2 channels are rapidly adapting mechanical-sensitive channels, they are sufficient to inspire sustained Ca2+ transients in Merkel cells, which further induce the release of neurotransmitters for tens of seconds. Thus our work provides a model that captures the membrane potentials and Ca2+ transients features of Merkel cells and partly explains how Merkel cells transduce the mechanical stimuli by Piezo2 channels.

In this demonstration, we suggest lightweight haptic communications from the context of remote neurorehabilitation. In particular, we collect tactile data from pressure sensors attached to the human hand and design a classifier to determine the objects that an individual holds or grasps. Finally, we have implemented the proposed system on Raspberry Pi and demonstrated a personalized classification while rendering the haptic feedback in a virtual perception.

In this demonstration, we present a wearable haptic system to realize the perceptual illusion in a virtual environment. We collect the pressure data from sensors attached with the fingertip, and after passing through a classifier, we render the sensation of pressure. We mainly focus on designing a lightweight and wearable haptic interface with fast rendering. At last, we implement the proposed interface on Raspberry Pi and present preliminary results with various test objects.

Depth estimation from single monocular image attracts increasing attention in autonomous driving and computer vision. While most existing approaches regress depth values or classify depth labels based on features extracted from limited image area, the resulting depth maps are still perceptually unsatisfying. Neither local context nor low-level semantic information is sufficient to predict depth. Learning based approaches suffer from inherent defects of supervision signals. This paper addresses monocular depth estimation with a general information exchange convolutional neural network. We maintain a high-resolution prediction throughout the network. Meanwhile, both low-resolution features capturing long-range context and fine-grained features describing local context can be refined with information exchange path stage by stage. Mutual channel attention mechanism is applied to emphasize interdependent feature maps and improve the feature representation of specific semantics. The network is trained under the supervision of improved log-cosh and gradient constraints so that the abnormal predictions have less impacts and the estimation can be consistent in high order. The results of ablation studies verify the efficiency of every proposed components. Experiments on the popular indoor and street-view datasets show competitive results compared with the recent state-of-the-art approaches.

With the aim of addressing the difficulty in identifying temperatures in virtual chemistry experiments, we propose a temperature-sensing simulation method of virtual chemistry experiments. We construct a virtual chemistry experiment temperature simulation platform, based on which a wearable temperature generation device is developed. The typical middle school virtual experiments of concentrated sulfuric acid dilution and ammonium nitrate dissolution are conducted to verify the actual effect of the device. The platform is capable to indicate near real-world experimental situations. The performance of the device not only meets the temperature sensing characteristics of human skin, but also matches the temperature change of virtual chemistry experiments in real-time. It is demonstrated that this temperature-sensing simulation method can represent exothermic or endothermic chemistry experiments, which is beneficial for students to gain understanding of the principles of thermal energy transformation in chemical reactions, thus avoiding the danger that may be posed in the course of traditional teaching of chemistry experiments effectively. Although this methoddoes not have a convenient enough operation for users, the immersion of virtual chemical experiments can be enhanced.

Predicting depth from single monocular image is a challenging task in scene understanding. Most existing work predicts depth by regression or classification with features extracted from local neighborhood area. However, neither regression nor classification achieves the final satisfying solution and local context can be insufficient to predict the depth. This paper innovatively addresses this problem as regression of class related features on a piled residual convolutional neural network. Our framework works at two stages. First, a well-designed deep convolutional neural network model is employed to classify the depths in difference-scale invariance space. The model utilizes all scales of context though piled residual paths. The deeper layers that capture high-level semantic features with long-range context can be directly refined using fine-grained features with local context from earlier convolutions. We then apply centered information gain loss to the model to produce intra-class compact and inter-class discriminative features. Second, to obtain depths instead of class labels, we infer depth regression with convolutional layers which model the mapping from class discriminative features to continuous depth values. Experiments on the popular indoor and outdoor datasets show competitive results compared with the recent state of the art methods.

This paper explores new approaches to recognize Chinese characters in images through touch sense. We extract the gradient image fused with brightness and chrominance information by the flow-based Gauss difference algorithm, and use the edge tangential flow method to smooth image contour, to obtain smooth character contours with significant edge features. Then, we use a tactile generation algorithm with five mechanical elements to generate stable and continuous tactile information based on the normal vector of triangular patches of the surface. To verify the feasibility and effectiveness of the proposed methods, we developed a Chinese character tactile sensing system. Experiments show that Chinese characters in images can be recognized by touch sense with higher recognition accuracy rate.

Force generation will improve the immersion and authenticity of the virtual environment with human computer interaction. Considering the limitations of single-point interaction with force generation, this paper proposes real-time force generation methods based on double-points interaction. Analyzing the double-points interaction behaviors, we propose different equations to calculate the interactive force under four different interaction states. We establish an experimental environment to test the efficiency of double-points haptic interaction. The evaluation results show that the method can generate interactive force in time, enhance the immersion and authenticity of the virtual environment, and improve the naturalness of human computer interaction.

Finger workspace is one of the most important features to evaluate the kinematic performances of the multifinger hands. It is also benefit for grasp planning, mechanical design, and motion control. Although many multi-finger hands have been developed over the last few decades, how to generate and visualize their workspaces is still a challengeable problem. We propose a workspace generation and visualization approach to explain the workspace characteristics of a multi-finger hand (ZSTU hand), which can help to solve the problem of choosing a grasp with the gripper constrains. Compared with previous researches, this approach can model more integrated fingertip workspaces with vivid forms and contours, which is valuable for researchers to accurately and intuitionally understand the fingertip motion ranges and the analysis of the workspace volume. These performances can help researchers to optimize the mechanical structures of the multi-finger hand, and improve the kinematic performances of the multi-finger hand.

The paper proposes a method of haptic rendering for virtual hand moving an object in virtual environments, which would not only improve the immersion and authenticity of the virtual reality system, but also help operators predict virtual objects' natural behavior and direct their interaction with virtual environments. By tracking the position of the grasped object, the posture change of the grasped object is detected. An algorithm is used to calculate the distribution of the external force imposed on the grasped object in arbitrary posture, and the static grasp force of the virtual hand is regenerated according to the physically-based general force model of virtual hand grasp. Based on the theories of kinematics and dynamics, the resultant force which causes the variation in motion of the grasped object can be obtained. Further, the grasp force of the virtual hand is regenerated by combining the posture change and the resultant force. The solution has been experimentally implemented using an exoskeleton force-feedback glove. A special series of experimental results show that the static grasp force rendering approach for the virtual hand moving an object is computationally efficient while retaining a well level of realism.

Accurate measurement of lower limb alignment is vital for orthopedic surgery planning. Based on the 3D lower limb bone models reconstructed from CT images, a 3D feature construction system is developed to find the lower limb alignment. This paper presents new methods to automatically identify the anatomic axes, mechanical axes, joint center points, and reference planes of the lower extremities. As far as we know, there is no other system that can construct all these features automatically. With the above identified features, our system can calculate the related angles of the lower limb alignment accurately for orthopedic knee surgery planning. We applied our system on two groups of specimen (with 6 pairs of lower limbs), and the resulting related angles are identical to the actual angles of the lower limb alignment.