We introduce Reality Fusion, a novel robot teleoperation system that localizes, streams, projects, and merges a typical onboard depth sensor with a photorealistic, high resolution, high framerate, and wide FoV rendering of the complex remote environment represented as 3D Gaussian splats (3DGS). Our framework enables robust egocentric and exocentric robot teleoperation in immersive VR, with the 3DGS effectively extending spatial information of a depth sensor with limited FoV and balancing the trade-off between data streaming costs and data visual quality. We evaluated our framework through a user study with 24 participants, which revealed that Reality Fusion leads to significantly better user performance, situation awareness, and user preferences. To support further research and development, we provide an open-source implementation with an easy-to-replicate custom-made telepresence robot, a high-performance virtual reality 3DGS renderer, and an immersive robot control package.1

We present mixed reality (MR) tunneling, a novel method to balance the trade-off between limited render performance and high visual quality of video see-through head-mounted displays (VST-HMDs) through fusing images of two types of camera sensors with different resolutions and frame rates. By merging a color video stream from an external stereoscopic camera with the grayscale VST commonly integrated into today’s standalone virtual reality (VR) headsets, we create a perceptually high-resolution and wide field of view VSTHMD prototype. The external high-resolution VST displayed at the central foveal to the para-peripheral region of the human visual field complements the low-resolution, low-latency grayscale VST at the far peripheral region, producing a tunneling effect, which simulates the human foveal and peripheral vision, with the potential to reduce cybersickness as in the tunneling effect in immersive VR. We propose two extensions to the MR tunneling method. The first one accommodates the user’s head movement speed by fading out the external VST when fast head movements are detected, thus potentially compensating for video streaming latency. The second one is a foveated MR tunneling effect, which displays the center of the external VST based on the tracked user eye movements. We evaluated the three MR tunneling methods in a within-subject study with 24 participants. The user study demonstrates the potential of our prototype and techniques based on the example of an assembly task that requires hand-eye coordination, untethered locomotion, and fine motor skills. The results demonstrate that, although not significant, the MR tunneling effects lead to higher overall usability, less perceived motion sickness, and a better sense of presence.1

Nowadays, high-power and multi-spectral lasers are used in many scientific experiments and industrial processes. Those laser sources can rapidly cause permanent damage to human eyes. Research and development work with those laser sources requires typically wearing personal protective equipment (PPE), such as laser safety goggles as eye protectors. Currently, laser safety goggles are based on optical spectral filters, which block spectral bands where hazardous laser radiation is emitted. Such laser safety goggles can filter up to 99% of the visible spectrum, rendering researchers working in hazardous and complex laboratory environments visually impaired. Video see-through head-mounted displays (VST-HMD) could be used as eye protectors without reducing users’ visibility of the environment since they can be constructed such that all laser and ambient light is blocked from the human eye. To date, this application domain is still largely unexplored in the MR community. To our best knowledge, there has been no comprehensive work that investigates the human factors of such an eye protection method at an advanced optics laboratory. In this work, we present the results of an empirical study where we evaluate the usability, perceived safety, advantages, and limitations of using VST-HMDs as laser safety goggles. We use a stereoscopic VST-HMD developed through a human-centered design approach at one of the most advanced optics laboratories in the world. 18 participants including 14 laser experts evaluated the current prototype. Our user evaluation and field studies confirm that the complex and hazardous working conditions at high-energy laser laboratories could be significantly improved with MR technology.