To assist robots in teleoperation tasks, haptic rendering which allows human operators access a virtual touch feeling has been developed in recent years. Most previous haptic rendering methods strongly rely on data collected by tactile sensors. However, tactile data is not widely available for robots due to their limited reachable space and the restrictions of tactile sensors. To eliminate the need for tactile data, in this paper we propose a novel method named as Vis2Hap to generate haptic rendering from visual inputs that can be obtained from a distance without physical interaction. We take the surface texture of objects as key cues to be conveyed to the human operator. To this end, a generative model is designed to simulate the roughness and slipperiness of the object's surface. To embed haptic cues in Vis2Hap, we use height maps from tactile sensors and spectrograms from friction coefficients as the intermediate outputs of the generative model. Once Vis2Hap is trained, it can be used to generate height maps and spectrograms of new surface textures, from which a friction image can be obtained and displayed on a haptic display. The user study demonstrates that our proposed Vis2Hap method enables users to access a realistic haptic feeling similar to that of physical objects. The proposed vision-based haptic rendering has the potential to enhance human operators' perception of the remote environment and facilitate robotic manipulation.

Robotic hand exoskeletons can provide assistance to people who suffer from hand functional disability or spinal cord injury (SCI). However, the current hand exoskeletons remain challenging with respect to having a user-friendly design that satisfies human motion with a lightweight structure. Here we propose a method of using topology optimization in the design of finger exoskeletons, which is a lightweight and integrate manufactured exoskeleton. The exoskeleton is designed by generating the topology configuration according to the bending state of the finger. The objective function in the optimization is set to be the maximization of output displacement of the design domain. After obtaining the topology configuration, a conjugate surface flexure hinge is developed as an elastic joint to replace the compliant part of the topologies. Finally, a rigid-compliant parallel exoskeleton is fabricated by a 3D printer and the performance of the mechanism is verified.

Many of current human pose estimation methods based on depth images require training stage. However, the training stage costs huge work on making samples. And many methods for human pose occlusion condition cannot work well. In this study, a novel approach to estimate human pose with a depth image called model‐based recursive matching (MRM) is introduced. A human skeleton model with customised parameters is created based on T‐pose to fit different body types. The authors use depth image and 3D point cloud corresponding to input. In contrast to previous work, the proposed method avoids training step and can give an accurate estimation in the case of the human occlusion condition. They demonstrate the method by comparing to the method Kinect offered by using random forest on 20 human poses. And the ground truth of coordinates of pose joint is made by the motion capture system. The result shows that the proposed method not only works well on the general human pose but also can deal with human occlusion better. And the authors’ method can be also applied to the disabled people and other creatures.

Current surgical tele-manipulators do not provide explicit haptic feedback during soft tissue palpation. Haptic information could improve the clinical outcomes significantly and help to detect hard inclusions within soft-tissue organs indicating potential abnormalities. However, system instability is often caught by direct force feedback. In this paper, a new approach to intra-operative tumor localization is introduced. A virtual-environment tissue model is created based on the reconstructed surface of a soft-tissue organ using a Kinect depth sensor and the organ's stiffness distribution acquired during rolling indentation measurements. Palpation applied to this tissue model is haptically fed back to the user. In contrast to previous work, our method avoids the control issues inherent to systems that provide direct force feedback. We demonstrate the feasibility of this method by evaluating the performance of our tumor localization method on a soft tissue phantom containing buried stiff nodules. Results show that participants can identify the embedded tumors; the proposed method performed nearly as well as manual palpation.

Advanced tactile tools in minimally invasive surgery have become a pressing need in order to reduce time and improve accuracy in localizing potential tissue abnormalities. In this regard, one of the main challenges is to be able to estimate tissue parameters in real time. In palpation, tactile information felt at a given location is identified by the viscoelastic dynamics of the neighboring tissue. Due to this reason the tissue examination behavior and the distribution of viscoelastic parameters in tissue should be considered in conjunction. This paper investigates the salient features of palpation behavior on soft tissue determining the effectiveness of localizing hard nodules. Experimental studies involving human participants, and validation tests using finite element simulations and a tele-manipulator, were carried out. Two distinctive tissue examination strategies in force-velocity modulation for the given properties of target tissue were found. Experimental results suggest that force-velocity modulations during continuous path measurements are playing an important role in the process of mechanical soft tissue examination. These behavioral insights, validated by detailed numerical models and robotic experimentations shed light on future designs of optimal robotic palpation.

Robot-assisted minimal invasive surgery made it possible to improve the quality of surgical procedures and to enhance clinical outcomes. However, the need to palpate soft tissue organs with the aim to localize potential sites of abnormalities in real time has been recognized. For this work, ten subjects were recruited to perform a remote palpation procedure on a silicone phantom utilizing a tele-manipulation setup, to study their behavior when remotely palpating soft tissue. The stiffness values acquired during the remote palpation of a silicone phantom were transferred to the subjects by means of haptic and visual feedback. Participating subjects were asked to detect hard nodules in the silicone tissue using two distinct strategies: a) randomly chosen movements, and b) trajectory pattern, based on manual palpation techniques for clinical breast examination. We have compared relevant parameters, defining patterns observed during manual palpation, with the counterpart patterns occurring during remote palpation. The results show the effectiveness of applying palpation trajectory pattern used during manual soft tissue examination to tele-manipulation palpation.

During open surgery, surgeons can perceive the locations of tumors inside soft-tissue organs using their fingers. Palpating an organ, surgeons acquire distributed pressure (tactile) information that can be interpreted as stiffness distribution across the organ -an important aid in detecting buried tumors in otherwise healthy tissue. Previous research has focused on haptic systems to feedback the tactile sensation experienced during palpation to the surgeon during minimally invasive. However, the control complexity and high cost of tactile actuators limits its current application. This paper describes a pneumatic multi-fingered haptic feedback system for robot-assisted minimally invasive surgery. It simulates soft tissue stiffness by changing the pressure of an air balloon and recreates the deformation of fingers as experienced during palpation. The pneumatic haptic feedback actuator is validated by using finite element analysis. The results prove that the interaction stress between the fingertip and the soft tissue as well as the deformation of fingertips during palpation can be recreated by using our pneumatic multi-fingered haptic feedback method.

This paper introduces a new and low-cost tissue stiffness simulation technique for surgical training and robot-assisted minimally invasive surgery (RMIS) with pseudo-haptic feedback based on tissue stiffness maps provided by rolling mechanical imaging. Superficial palpation and deep palpation pseudo-haptic simulation methods are presented. Although without expensive haptic interfaces users receive only visual feedback (pseudo-haptics) when maneuvering a cursor over the surface of a virtual soft-tissue organ by means of an input device such as a mouse, a joystick, or a touch-sensitive tablet, the alterations to the cursor behavior induced by the method creates the experience of actual interaction with a tumor in the users' minds. The proposed methods are experimentally evaluated for tissue abnormality identification. It is shown that users can recognize tumors with these two methods and the rate of correctly recognized tumors in deep palpation pseudo-haptic simulation is higher than superficial palpation simulation.