Teleoperated techniques enable remote human-robot interaction and have been widely accepted in robot-assisted surgeries. However, it is still hard to guarantee the safety of teleoperated surgery due to the imperfect input commands limited by remote perception, preventing teleoperated surgery from being widely used. We propose a new framework to avoid the collision of surgery robots and human tissue caused by inaccurate inputs. We directly take the medical volume data and propose to use the implicit neural field to guide teleoperated robot-assisted surgery. With guidance, the trajectory of the robot manipulator is optimized to safely work inside a narrow workspace. We evaluated our method in several aspects and conducted a real-world experiment on a head phantom. Experimental results show that our proposed method can effectively avoid the collision between the surgical tool and the human tissue during teleoperation.

The neural radiance field (NeRF) is attracting increasing attentions from researchers in various fields. While NeRF has produced visually plausible results and found its potential applications in virtual reality, users are only allowed to rotate the camera to observe the scene represented as NeRF. We study the haptic interaction with NeRF models in this paper to enable the experience of touching objects reconstructed by NeRF. Existing haptic rendering algorithms do not work well for NeRF-represented models because NeRF is often noisy. We propose a stochastic haptic rendering method to deal with the collision response between the haptic proxy and NeRF. We validate our method with complex NeRF models and experimental results show the efficacy of our proposed algorithm.

We propose a novel approach for enabling comic-style conversation in mixed reality to assist face-to-face conversation on-site or remotely. Our approach brings word balloons of comic-style conversation to the real world. The word balloons can adapt to mixed reality scenes, such as the 3-D head motion of the speaker, the comic styles, and the speech. During the conversation, our approach updates the word balloons continuously in the object space and discretely in the image space, guided by a field learned from comics. Quantitative experiments and perceptual studies were conducted to evaluate and compare our approach with alternatives. The results from the user study and ablation study demonstrated that our approach turns out to be practical for assisting face-to-face conversation.

Haptic feedback is widely used to enhance realism in virtual reality (VR). Shape and softness are two common factors perceived by the users in the haptic rendering of soft objects. To integrate these factors, we propose a new handheld shape-changing device, TapeTouch, to provide various shapes and softness in real time. TapeTouch is based on a controllable shape-changing tape, which is mainly composed of four motors and a section of brass tape. We design a structure of the components to fit a portable controller and allow to flexibly adjust the shape of the brass tape. After decoding desired shapes into the signals to control the motor, we automatically reproduce varying shapes and levels of softness to the finger or palm touching the shape-changing tape. We conducted two user studies to understand the capability of TapeTouch to render shape and softness, and the results showed that TapeTouch could provide a variety of distinguishable shapes as well as multiple levels of softness. Based on the results, we performed two VR experience studies to verify that the haptic feedback from TapeTouch enhances VR realism.