Augmented reality (AR) has great potential for use in healthcare applications, especially remote medical training and supervision. In this paper, we analyze the usage of an AR communication system to teach a medical procedure, the placement of a central venous catheter (CVC) under ultrasound guidance. We examine various AR communication and collaboration components, including gestural communication, volumetric information, annotations, augmented objects, and augmented screens. We compare how teaching in AR differs from teaching through videoconferencing-based communication. Our results include a detailed medical training steps analysis in which we compare how verbal and visual communication differs between video and AR training. We identify procedural steps in which medical experts give visual instructions utilizing AR components. We examine the change in AR usage and interaction over time and recognize patterns between users. Moreover, AR design recommendations are given based on post-training interviews.

We describe three interactive augmented reality stories for children that we showed at the Cannes Film Festival “Marche du Film” in July 2021. The stories were developed using a novel technique: 3D modeling and animation from within the Virtual Reality. The audience at Cannes viewed and interacted with these stories using mixed reality glasses, a prototype of the Sony Spatial Display, and an AR-enabled tablet. We report on the technical development process and the feedback from the Cannes audience.

We design a volumetric communication system for remote assistance of procedural medical tasks. The system allows a remote expert to spatially guide a local operator using a real-time volumetric representation of the patient. Guidance is provided by voice, virtual hand metaphor, and annotations performed in situ. We include the feedback we received from the medical professionals and early NASA TLX [5] data on the cognitive load of the system.

We present “The Meeting”, an immersive volumetric theatre experience in mixed reality. The user takes on the role of Vera, a woman reconnecting with a former partner, Robert. The actor playing Robert was volumetrically captured. The user can move around the “stage”, performing lines of dialogue using prompting or in-air text. Robert reacts to users’ utterances accordingly. “The Meeting” is one of the first interactive theatre plays that uses volumetric capture technology in an interactive manner and places the actors in an interactive mixed-reality space.

We present our work in progress, a real-time mixed reality communication system for remote assistance in medical emergency situations. 3D cameras capture the emergency situa-tion and send volumetric data to a remote expert. The remote expert sees the volumetric scene through mixed reality glasses and guides an operator at the patient. The local operator receives audio and visual guidance augmented onto the mixed reality headset. We compare the mixed reality system against traditional video communication in a user study on a CPR emergency simulation. We evaluate task performance, cognitive load, and user interaction. The results will help to better understand the benefits of using augmented and volumetric information in medical emergency procedures.

Medical procedures are an essential part of healthcare delivery, and the acquisition of procedural skills is a critical component of medical education. Unfortunately, procedural skill is not evenly distributed among medical providers. Skills may vary within departments or institutions, and across geographic regions, depending on the provider’s training and ongoing experience. We present a mixed reality real-time communication system to increase access to procedural skill training and to improve remote emergency assistance. Our system allows a remote expert to guide a local operator through a medical procedure. RGBD cameras capture a volumetric view of the local scene including the patient, the operator, and the medical equipment. The volumetric capture is augmented onto the remote expert’s view to allow the expert to spatially guide the local operator using visual and verbal instructions. We evaluated our mixed reality communication system in a study in which experts teach the ultrasound-guided placement of a central venous catheter (CVC) to students in a simulation setting. The study compares state-of-the-art video communication against our system. The results indicate that our system enhances and offers new possibilities for visual communication compared to video teleconference-based training.

“Dill Pickle”, is the first interactive immersive theatre experience in virtual reality that uses volumetric capture. In the play, a volumetrically captured actor plays the character of Robert. The user interacts with Robert through utterances that are memorized or prompted with text or audio. The set was recreated through a process of photogrammetry.

In this study, we investigated the effect of augmented reality aided dashboard information projected on the windshield on car driving performance. As augmented reality technology becomes more popular and accessible, essential driving information such as the current velocity can be projected to where the traffic happens in terms of location and distance. In comparison with a Heads Up Display (HUD) on a windshield, augmented reality devices offer the possibility to place the data farther in front of the car. Because of this, the eyes of the driver do not need to re-focus constantly to a closer object, when most of the imminent attention needs to be set to the space in front of the vehicle. To test in a safe environment, we developed a virtual reality car parkour and conducted a user study. Splitting our participants in two equally big groups, one with a conservative dashboard and the other with the velocity displayed 10 m ahead and over the horizon. Assessing the participant's performance in areas such as maintaining the speed limit, distance to the traffic, and time to match up with the given speed limit, we found striking positive results using the augmented reality solution.

In real life, people communicate using both speech and non-verbal signals such as gestures, face expression or body pose. Non-verbal signals impact the meaning of the spoken utterance in an abundance of ways. An absence of non-verbal signals impoverishes the process of communication. Yet, when users are represented as avatars, it is difficult to translate non-verbal signals along with the speech into the virtual world without specialized motion-capture hardware. In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of Generative Adversarial Neural Networks (GANs) to model the correlation rather than causation between speech and gestures. This approach approximates neuroscience findings on how non-verbal communication and speech are correlated. We create a large dataset which consists of speech and corresponding gestures in a 3D human pose format from which our model learns the speaker-specific correlation. We evaluate the proposed technique in a user study that is inspired by the Turing test. For the study, we animate the generated gestures on a virtual character. We find that users are not able to distinguish between the generated and the recorded gestures. Moreover, users are able to identify our synthesized gestures as related or not related to a given utterance.

Volumetric filmmaking is a novel and inherently interactive medium. In volumetric film the viewer takes over the director's responsibility for selecting the point of view from which the story is being told. The viewer becomes the cinematographer and the editor of the film at the moment of viewing. In this paper, we provide an overview of interaction modes in volumetric film and compare volumetric film to both traditional film and 360 video.

This work explores the role of immersive learning in regards to predicting molecular structures and orbital hybridization through an Oculus Rift HMD. Specifically, this research utilizes an interactive virtual environment to help teach the concept of the Valence Shell Electron Pair Repulsion Theory. We developed an interactive and immersive simulation in which participants constructed virtual molecules. Following the simulation, participants were asked to identify shapes of similar molecules to measure their understanding of the theory as compared to a control group. Finally, participants answered the Game Experience Questionnaire to assess the level of Immersion. The results show that our immersive virtual reality (IVR) experience improves comprehension of three dimensional structures in chemistry.

In this work we present our approach of measuring the influence, importance and immersion of haptic feedback in a virtual reality learning environment. Participants were given the goal to improve themselves to their best ability on a task involving virtual weights supported by real world objects in a mixed reality environment. Conducting pre- and post-surveys, assessing the immersion and putting the resulting data in comparison with the test group, which used a pure virtual reality environment. We found correlations bolstering our initial assumption of the importance of haptic feedback in understanding and usage of virtual reality learning environments for specific functions.

In this work we are presenting a novel virtual learning environment for understanding the phenomena of gravitational waves. Emphasizing on the popular aspect of this topic, we developed an interactive and immersive gravitational wave simulation using a field density representation. We identified three main areas of understanding gravitational waves such as wave source, spatial irradiation distribution and wave type, which the virtual learning environment supports and was designed for. Participant were tested on their knowledge post and prior to the VLE experience as well as on their perceived immersion with the Game Experience Questionnaire. We found striking results in improving the understanding for gravitational waves, that underlines the feasibility of your virtual learning environment.

Volumetric capture is a technique that allows to create “holographic” recordings of actors, sets and props. The technique can be used to create immersive stories that sometimes reflect aspects of reality better than realistic 3D models. For example, volumetric captures of actors do not seem to cause uncanny valley effect. In this paper, we provide an overview of volumetric capture technology and its application to narrative filmmaking.

In this project we created an expandable framework for allowing infinite walking in virtual reality in a closed play area. A saccade is a rapid eye movement with a unique property: the eye temporarily gathers reduced information – saccadic suppression. We leverage the suppression to redirect the user’s walking towards the center of the play area by rotating the virtual world around the camera’s location. With the VR environment and corresponding pre and post experience questions we could already show an improvement in understanding on a set of participants. Modern VR hardware such as the Vive Eye Pro allows a reasonable sample rate of eye movement measurements. A self-developed VR testing environment was used and with corresponding pre and post experience questions we tested a group of participants regarding general motion and VR- sickness parameters. We found a certain angle for the maximum saccade rotation which was base of further testing. We found, that our framework and the default settings successfully allow saccadic redirection with only marginal discomfort for the users.

While many metaphors were developed for interactions from a specific point at the reality-virtuality continuum, much less attention has been paid to designing metaphors that allow the users to cross the boundaries between the virtual, the augmented, and the real. We propose a use of an Inter-Reality Interactive Surface (IRIS) that enables users to collaborate across the reality-virtuality continuum within the same application. While we examine IRIS in the context of an immersive educational platform, UniVResity, the metaphor can be generalized to many other application domains.

We describe a prototype of the virtual reality remote classroom participation system called UniVResity. UniVResity mirrors in virtual reality the ongoing face-to-face classroom activities, taking into account potentially low bandwidth data connection and lack of VR equipment in class. Our system attempts to combine the benefits of online education and face-to-face education, and makes face-to-face learning more accessible.

We present “AHMED”, a mixed-reality toolset that allows visitors to experience mixed-reality museum or art exhibitions created ad-hoc at locations such as event venues, private parties,or a living room. The system democratizes access to exhibitions for populations that cannot visit these exhibitions in person for reasons of disability, time-constraints, travel restrictions, or socio-economic status.

We present the design and evaluation of Scalebridge VR, an immersive educational game that teaches children the mathematical skill of proportional reasoning. The game uses brain-computer-interface-based adaptive level difficulty to modulate difficulty of the game based on the player’s attention and meditation state. The game is an adaptation of previously introduced Scalebridge game that did not use virtual reality, but was shown to be an effective tool for learning proportional reasoning.

We present "Hamlet", a prototype implementation of a virtual reality experience in which a player takes on a role of the theater director. The objective of the experience is to direct Adam, a virtual actor, to deliver the best possible performance of Hamlet's famous "To be, or not to be" soliloquy. The player interacts with Adam using voice commands, gestures, and body motion. Adam responds to acting directions, offers his own interpretations of the soliloquy, acquires the choreography from the player's body motion, and learns the scene blocking by following the player's pointing gestures.

We present Gauntlet, a travel technique for immersive environments that uses non-dominant hand tracking and a fist gesture to translate and rotate the viewport. The technique allows for simultaneous use of the dominant hand for other spatial input tasks. Applications of Gauntlet include FPS games, and other application domains where navigation should be performed together with other tasks. We release the technique along with an example application, a VR horror game, as an open source project.

We present an animation synthesis technique that produces high quality avatar animation from coarse or incomplete motion input by matching, in real-time, the input against thousands of high-quality pre-recorded motion capture templates. The technique uses subsequence multivariate Quaternion Dynamic Time Warping with the angle between quaternions as a distance measure. Our implementation of the technique using compute shaders shows performance gains orders of magnitude greater than the state-of-the-art DTW implementations.

Due to an increase in the prevalence and intensity of wildfires worldwide [Liu et al. 2010], it is becoming more important to understand campfire safety in order to prevent human-caused wildfires. In the United States, the most common fire safety advice comes in the form of commercials and posters as a part of the Smokey the Bear campaign [Smo 2017]. Presenting this information through a virtual reality game provides a controlled and engaging environment to practice and learn how to safely control a campfire. This immersive experience guides the user through every step of creating and extinguishing a campfire based on information from the Smokey the Bear campaign. VR Wildfire Prevention aims to engage and educate people in campfire safety by providing a controlled environment to practice the relevant techniques while incentivizing proper behavior through gamification. Players of the game report that the game is an enjoyable experience.

A smartphone-based head mounted display combined with a motion tracking device provides an affordable wireless interactive virtual reality system. A common problem of such a setup is high end-to-end latency. We evaluate low-cost motion trajectory prediction techniques and show that latency compensation can be realized without using much of the smartphone's computational resources. We apply the technique to reduce motion tracking lag in a smartphone-based virtual reality lightsaber fighting game.

We describe the design and evaluation of a freehand, smartwatch-based, mid-air pointing and clicking interaction technique, called Watchpoint. Watchpoint enables a user to point at a target on a nearby large display by moving their arm. It also enables target selection through a wrist rotation gesture. We validate the use of Watchpoint by comparing its performance with two existing techniques: Myopoint, which uses a specialized forearm mounted motion sensor, and a camera-based (Vicon) motion capture system. We show that Watchpoint is statistically comparable in speed and error rate to both systems and, in fact, outperforms in terms of error rate for small (high Fitts's ID) targets. Our work demonstrates that a commodity smartwatch can serve as an effective pointing device in ubiquitous display environments.

TickTockRay is a smartwatch-based raycasting technique designed for smartphone-based head mounted displays. It demonstrates that smartwatch-based raycasting can be reliably implemented on an off-the-shelf smartphone and may provide a feasible alternative for specialized input devices. We release TickTockRay to the research community as an open-source plugin for Unity along with an example application, a Minecraft VR game clone, that shows the utility of the technique for placement and destruction of Minecraft blocks.

We develop and formally evaluate a metaphor for smartphone interaction with 3D environments: Tiltcasting. Under the Tiltcasting metaphor, users interact within a rotatable 2D plane that is "cast" from their phone's interactive display into 3D space. Through an empirical validation, we show that Tiltcasting supports efficient pointing, interaction with occluded objects, disambiguation between nearby objects, and object selection and manipulation in fully addressable 3D space. Our technique out-performs existing target agnostic pointing implementations, and approaches the performance of physical pointing with an off-the-shelf smartphone.

Classroom response systems (CRS) have been shown to dramatically improve a variety of learning outcomes in science and engineering education. Their adoption in research university classrooms has been very slow. This paper postulates a contributing factor to the lack of widespread adoption of CRS in that context and introduces a new CRS designed to mitigate the identified factor. The CRS, NetClick, is built on the premise that the requirement of authoring new content prevents many Professors from using existing CRS. After describing the operation and advantages of NetClick to other CRS, we describe initial feedback from Computer Science Professors on its use for converting existing teaching slides into interactive content.

We propose spatial correspondence targeting to support interaction between devices in multi-device environments when network connectivity fails. In spatial correspondence targeting, for a given target on surface A, an end-user envisions the relative position of that target on surface B and interacts on surface B without any visual depiction of the target on surface B. The targeting task relies on human spatial visualization ability, i.e. the ability to relate the spatial position of objects on one display to their scale-invariant position on another display. We provide experimental evidence that demonstrates that users may be able to target up to 25 discrete targets using a smartphone screen even in the absence of a depiction of the target on the smartphone screen. We argue that the accuracy of spatial correspondence targeting is sufficient for the technique to have many practical applications.

Feature-based model templates have been recently proposed as a approach for modeling software product lines. Unfortunately, templates are notoriously prone to errors that may go unnoticed for long time. This is because such an error is usually exhibited for some configurations only, and testing all configurations is typically not feasible in practice. In this paper, we present an automated verification procedure for ensuring that no ill-structured template instance will be generated from a correct configuration. We present the formal underpinnings of our proposed approach, analyze its complexity, and demonstrate its practical feasibility through a prototype implementation.