In virtual reality (VR), using the hands instead of controllers has enormous potential for intuitive user interactions, e.g., direct grasping, touching, or moving virtual objects. In this context, the realism of the hands plays an important role in the body ownership illusion and the user’s feeling of presence. Yet, most applications use generic 3D hand models, so they do not include individual user characteristics, like wrinkles, moles, or skin features. In this work, we present a pipeline to include fully articulated, photorealistic, fully tracked hand scans for VR applications. Our pipeline captures photos of the hand from different perspectives simultaneously using a customized hand scanner. Using open-source photogrammetry software, a rigged 3D model of the hand is created and integrated into a Unity application. At last, we provide practitioners and researchers with a fast and easy way to integrate a user’s real hands into a VR application. Finally, we present guidelines for generating a fully articulated, personalized virtual hand model within 30 minutes from a set of 2D images.

The development of radiance fields (RF), such as 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF), has revolutionized interactive photorealistic view synthesis and presents enormous opportunities for XR research and applications. However, despite the exponential growth of RF research, RF-related contributions to the XR community remain sparse. To better understand this research gap, we performed a systematic survey of current RF literature to analyze (i) how RF is envisioned for XR applications, (ii) how they have already been implemented, and (iii) the remaining research gaps. We collected 365 RF contributions related to XR from computer vision, computer graphics, robotics, multimedia, human-computer interaction, and XR communities, seeking to answer the above research questions. Among the 365 papers, we performed an analysis of 66 papers that already addressed a detailed aspect of RF research for XR. With this survey, we extended and positioned XR-specific RF research topics in the broader RF research field and provide a helpful resource for the XR community to navigate within the rapid development of RF research.

The increasing relevance of artificial intelligence (AI) agents for people’s everyday professional and private lives leads to a great need for identifying efficient and user-friendly modalities for human-agent interaction. Such interaction usually takes place via text or voice. While chatbots and voice assistants each rely on only one modality, the focus of this paper is on asymmetric modality combinations. A user study was conducted in the context of information retrieval, and different input and output modalities were evaluated in a 2 × 2 within-subject design. It was found that efficient modality combinations are not always preferred and that usability factors, such as user control, have a greater influence on preference. For example, the combination of voice input and text output can prove to be very efficient, but at the same time be less preferred due to weaknesses in the user experience. Pure text-based interaction was found to be the preferred combination as it ensures a high degree of control and freedom for the user. Dialog-based applications for information retrieval should therefore use the text modality as a basis and supplement this with voice-based input and output options.

We introduce Reality Fusion, a novel robot teleoperation system that localizes, streams, projects, and merges a typical onboard depth sensor with a photorealistic, high resolution, high framerate, and wide FoV rendering of the complex remote environment represented as 3D Gaussian splats (3DGS). Our framework enables robust egocentric and exocentric robot teleoperation in immersive VR, with the 3DGS effectively extending spatial information of a depth sensor with limited FoV and balancing the trade-off between data streaming costs and data visual quality. We evaluated our framework through a user study with 24 participants, which revealed that Reality Fusion leads to significantly better user performance, situation awareness, and user preferences. To support further research and development, we provide an open-source implementation with an easy-to-replicate custom-made telepresence robot, a high-performance virtual reality 3DGS renderer, and an immersive robot control package.1

Uncertainty cues are inherent in natural human interaction, as they signal to communication partners how much they can rely on conveyed information. Humans subconsciously provide such signals both verbally (e.g., through expressions such as “maybe’’ or “I think’’) and non-verbally (e.g., by diverting their gaze). In contrast, artificial intelligence (AI)-based services and machine learning (ML) models such as ChatGPT usually do not disclose the reliability of answers to their users.

In modern spatial computing devices, users are confronted with diverse methods for object selection, including eye gaze (cf. Apple Vision Pro), hand gestures (cf. Microsoft HoloLens 2), touch gestures (cf. Google Glass Enterprise Edition 2), and external controllers (cf. Magic Leap 2). Although there are a plethora of empirical studies on which selection techniques perform best, a common limiting factor stems from the partly artificial setups. These typically exclude practical influences such as visual distraction.

Enabled by the latest achievements in artificial intelligence (AI), computer graphics as well as virtual, augmented, and mixed reality (VR/AR/MR), virtual agents are increasingly resembling humans in both their appearance and intelligent behavior. This results in enormous potential for agents to support users in their daily lives, for example in customer service, healthcare, education or the envisioned all-encompassing metaverse. Today's technology would allow users to customize their conversation partners in the metaverse - as opposed to reality - according to their preferences, potentially improving the user experience. On the other hand, there is little research on how reshaping the head of a communication partner might affect the immediate interaction with them. In this paper, we investigate the user requirements for and the effects of agent customization. In a two-stage user study ($N=30$), we collected both self-reported evaluations (e.g., intrinsic motivation) and interaction metrics (e.g., interaction duration and number of tried out items) for the process of agent customization itself as well as data on how users perceived the subsequent human-agent interaction in VR. Our results indicate that users only wish to have full customization for agents in their personal social circle, while for general services, a selection or even a definite assignment of pre-configured agents is sufficient. When customization is offered, attributes such as gender, clothing or hair are subjectively more relevant to users than facial features such as skin or eye color. Although the customization of human interaction partners is beyond our control, customization of virtual agents significantly increases perceived social presence as well as rapport and trust. Further findings on user motivation and agent diversity are discussed in the paper.

Real-time detection of saccades is of major interest for many applications in human-computer interaction and mixed reality. However, due to relatively low update rates and high latencies of current commercially available eye trackers, gaze events are typically detected after they occur with some delay. This limits interaction scenarios such as intent-based gaze interaction, redirected walking, or gaze forecasting.

Augmented Reality (AR) technology has been widely investigated to support various navigation tasks, including initial approaches that suggest its potential use on ships. For maritime navigation, skippers use a variety of information displayed on a ship’s bridge. However, the constant shift of focus between this information and the outside view of the ship might pose cognitive as well as safety challenges. Here, AR could facilitate the navigation of ships by overlaying the real-world view with spatially anchored visual navigation aids. Despite this potential, previous work mainly presents conceptual approaches, technical tests, or user studies performed in ship simulators only.In this paper, we evaluate an AR-based assistance system in the actual real-world water environment, where technical issues and varying physical conditions could influence the system’s usability. In collaboration with hydrographic experts following a user-centered design approach, a functional AR system was developed that virtually displays navigation aids on the water surface. In a field study, ten skippers used the system to navigate a ship along a path through a port area. We assessed the accuracy, perceived workload, and user experience of participants. In addition, qualitative feedback was thematically analyzed to retrieve insights about the skippers’ attitude regarding using AR on actual ships. We report lessons learned about aspects such as ergonomics, perceived safety challenges, as well as envisioned further use cases and extended data integration.

Medical virtual agents (VAs) hold great potential to support patients in achieving their health goals, especially at times or in regions where the demand for physiological and psychological therapy exceeds the capacity of medical services. To create an accepted complement to on-site diagnosis, treatment, and counseling, it is critical to understand the impact of factors such as the agent’s visual representation, behavior, and responsibilities on creating a trustworthy human-agent relationship. To gain insights into these factors, we conducted a systematic literature review including 59 papers on embodied VAs in the medical domain. Our review focused on the application fields and the role of VAs in medicine, as well as the technology used to display them. Using thematic analysis, we discuss our findings in terms of user preferences, as well as potentials and barriers faced in the interaction with medical VAs. Concerning the visual representation, the users’ wish for customization in terms of appearance and communication modalities was pointed out. It was also important that the agent’s information builds up on trustworthy sources, that they are motivating and adapted to the users’ knowledge. Finally, our results identify research gaps, in particular regarding the technological implementation and the use of artificial intelligence.

Recent advancements in Neural Radiance Fields (NeRF) provide enormous potential for a wide range of Mixed Reality (MR) applications. However, the applicability of NeRF to real-time MR systems is still largely limited by the rendering performance of NeRF. In this paper, we present a novel approach for Variable Rate Shading for Neural Radiance Fields (VRS-NeRF). In contrast to previous techniques, our approach does not require training multiple neural networks or re-training of already existing ones, but instead utilizes the raytracing properties of NeRF. This is achieved by merging rays depending on a variable shading rate, which reduces the overall number of queries to the neural network. We demonstrate the generalizability of our approach by implementing three alternative functions for the determination of the shading rate. The first method uses the gaze of users to effectively implement a foveated rendering technique in NeRF. For the other two techniques, we utilize shading rates based on edges and saliency. Based on a psychophysical experiment and multiple image-based metrics, we suggest a set of parameters for each technique, yielding an optimal tradeoff between rendering performance gain and perceived visual quality.

Raycasting-based interaction techniques are widely used for object selection in immersive environments. Despite their intuitive use, they come with challenges due to small or far away objects, hand tremor, and tracking inaccuracies. Previous adaptations for raycasting, such as directly snapping the ray to the closest target, extruding the ray to a cone, or multi-step selection techniques, require additional time for users to become familiar with them. To address these issues, we propose three assistive techniques in which the visible selection ray is subtly redirected towards a target, with a proximity and gain based increase in the redirection amount. In a user study (N = 26), we compared these redirection techniques with a baseline condition based on a Fitts’ law task and collected performance measures as well as comprehensive subjective feedback. The results indicate that the three redirection techniques are significantly faster and have higher effective throughput than the baseline condition. Participants retained a high sense of agency with all redirection techniques and reported significantly lower total workload compared to the baseline. The majority of participants preferred selection with assistive ray redirection and perceived it as not distracting or intrusive. Our findings support that assistive redirected raycasting techniques can improve object selection performance and user experience in virtual environments.

Initial motivation when starting exergaming is a key factor towards enabling long-term engagement and adherence, especially among older adults. To increase, in particular, the initial motivation of older adults, we introduce the concept of diminishing gaze assistance (GA), assess its feasibility for virtual reality (VR) exergames, and investigate the effects on motor learning, performance, and motivation in older adult users. First, we conducted a focus group followed by a pre-study on the development of VR exergames for older adults and VR gaze assistance. The results informed the design and implementation of our gaze-assisted throwing exergame, which was then evaluated in a follow-up main study. Participants of the main study were randomly assigned to the GA and Motor (control) group, and had to complete a VR throwing task, in which participants had to aim and throw at three targets at varying angles. The GA group received declining gaze assistance, in which the ball trajectory was initially guided by their gaze (rather than their physical (motor) throwing) before guidance was gradually reduced until their physical (motor) throwing ability was solely responsible for hitting the target. Motivation and user experience were assessed using the Questionnaire on Current Motivation before and during, and the short scale of intrinsic motivation questionnaire after the task. The results show that the GA was generally perceived positively. In particular, the initial confidence of the GA group was rated higher, and we observed evidence suggesting increased confidence throughout the trial.

Research on intelligent virtual agents (IVAs) often concerns the implementation of human-like behavior by integrating artificial intelligence algorithms. Thus far, few studies focused on mimicry of cognitive imperfections inherent to humans in IVAs. Neglecting to implement such imperfect behavior in IVAs might result in less believable or engaging human-agent interactions. In this paper, we simulate human imprecision in conversational IVAs’ temporal statements. We conducted a survey to identify temporal statement patterns, transferred them to a conversational IVA, and conducted a user study evaluating the effects of time precision on perceived anthropomorphism and usefulness. Statistical analyses reveal significant interaction between time precision and agents’ use of memory aids, indicating that (i) imprecise agents are perceived as more human-like than precise agents when responding immediately, and (ii) unnaturally high levels of temporal precision can be compensated for by memory aid use. Further findings underscore the value of continued inquiry into cultural variations.

We present mixed reality (MR) tunneling, a novel method to balance the trade-off between limited render performance and high visual quality of video see-through head-mounted displays (VST-HMDs) through fusing images of two types of camera sensors with different resolutions and frame rates. By merging a color video stream from an external stereoscopic camera with the grayscale VST commonly integrated into today’s standalone virtual reality (VR) headsets, we create a perceptually high-resolution and wide field of view VSTHMD prototype. The external high-resolution VST displayed at the central foveal to the para-peripheral region of the human visual field complements the low-resolution, low-latency grayscale VST at the far peripheral region, producing a tunneling effect, which simulates the human foveal and peripheral vision, with the potential to reduce cybersickness as in the tunneling effect in immersive VR. We propose two extensions to the MR tunneling method. The first one accommodates the user’s head movement speed by fading out the external VST when fast head movements are detected, thus potentially compensating for video streaming latency. The second one is a foveated MR tunneling effect, which displays the center of the external VST based on the tracked user eye movements. We evaluated the three MR tunneling methods in a within-subject study with 24 participants. The user study demonstrates the potential of our prototype and techniques based on the example of an assembly task that requires hand-eye coordination, untethered locomotion, and fine motor skills. The results demonstrate that, although not significant, the MR tunneling effects lead to higher overall usability, less perceived motion sickness, and a better sense of presence.1

Nowadays, high-power and multi-spectral lasers are used in many scientific experiments and industrial processes. Those laser sources can rapidly cause permanent damage to human eyes. Research and development work with those laser sources requires typically wearing personal protective equipment (PPE), such as laser safety goggles as eye protectors. Currently, laser safety goggles are based on optical spectral filters, which block spectral bands where hazardous laser radiation is emitted. Such laser safety goggles can filter up to 99% of the visible spectrum, rendering researchers working in hazardous and complex laboratory environments visually impaired. Video see-through head-mounted displays (VST-HMD) could be used as eye protectors without reducing users’ visibility of the environment since they can be constructed such that all laser and ambient light is blocked from the human eye. To date, this application domain is still largely unexplored in the MR community. To our best knowledge, there has been no comprehensive work that investigates the human factors of such an eye protection method at an advanced optics laboratory. In this work, we present the results of an empirical study where we evaluate the usability, perceived safety, advantages, and limitations of using VST-HMDs as laser safety goggles. We use a stereoscopic VST-HMD developed through a human-centered design approach at one of the most advanced optics laboratories in the world. 18 participants including 14 laser experts evaluated the current prototype. Our user evaluation and field studies confirm that the complex and hazardous working conditions at high-energy laser laboratories could be significantly improved with MR technology.

Natural interaction between multiple users within a shared virtual environment (VE) relies on each other's awareness of the current position of the interaction partners. This, however, cannot be warranted when users employ noncontinuous locomotion techniques, such as teleportation, which may cause confusion among bystanders. In this paper, we pursue two approaches to create a pleasant experience for both the moving user and the bystanders observing that movement. First, we will introduce a Smart Avatar system that delivers continuous full-body human representations for noncontinuous locomotion in shared virtual reality (VR) spaces. Smart Avatars imitate their assigned user's real-world movements when close-by and autonomously navigate to their user when the distance between them exceeds a certain threshold, i.e., after the user teleports. As part of the Smart Avatar system, we implemented four avatar transition techniques and compared them to conventional avatar locomotion in a user study, revealing significant positive effects on the observers' spatial awareness, as well as pragmatic and hedonic quality scores. Second, we introduce the concept of Stuttered Locomotion , which can be applied to any continuous locomotion method. By converting a continuous movement into short-interval teleport steps, we provide the merits of non-continuous locomotion for the moving user while observers can easily keep track of their path. Thus, while the experience for observers is similarly positive as with continuous motion, a user study confirmed that Stuttered Locomotion can significantly reduce the occurrence of cybersickness symptoms for the moving user, making it an attractive choice for shared VEs. We will discuss the potential of Smart Avatars and Stuttered Locomotion for shared VR experiences, both when applied individually and in combination.

Developers of interactive systems have a variety of interaction techniques to choose from, each with individual strengths and limitations in terms of the considered task, context, and users. While there are taxonomies for desktop, mobile, and virtual reality applications, augmented reality (AR) taxonomies have not been established yet. However, recent advances in immersive AR technology (i.e., head-worn or projection-based AR), such as the emergence of untethered headsets with integrated gesture and speech sensors, have enabled the inclusion of additional input modalities and, therefore, novel multimodal interaction methods have been introduced. To provide an overview of interaction techniques for current immersive AR systems, we conducted a literature review of publications between 2016 and 2021. Based on 44 relevant papers, we developed a comprehensive taxonomy focusing on two identified dimensions – task and modality. We further present an adaptation of an iterative taxonomy development method to the field of human-computer interaction. Finally, we discuss observed trends and implications for future work.

Avatars in virtual reality (VR) with fully articulated hands enable users to naturally interact with the virtual environment (VE). Interactions are often performed in a one-to-one mapping between the movements of the user’s real body, for instance, the hands, and the displayed body of the avatar. However, VR also allows manipulating this mapping to introduce non-isomorphic techniques. In this context, research on manipulations of virtual hand movements typically focuses on increasing the user’s interaction space to improve the overall efficiency of hand-based interactions.

In this paper we present a floor-based user interface (UI) that allows multiple users to explore a spatial augmented reality (SAR) environment with both monoscopic and stereoscopic projections. Such environments are characterized by a low level of user instrumentation and the capability of providing a shared interaction space for multiple users. However, projector-based systems using stereoscopic display are usually single-user setups, since they can provide the correct perspective for only one tracked person. To address this problem, we developed a set of guidance cues, which are projected onto the floor in order to assist multiple users regarding (i) the interaction with the SAR system, (ii) the identification of regions of interest and ideal viewpoints, and (iii) the collaboration with each other. In a user study with 40 participants all cues were evaluated and a set of feedback elements, which are essential to guarantee an intuitive self-explaining interaction, was identified. The results of the study also indicate that the developed UI guides users to more favorable viewpoints and therefore is able to improve the experience in a multi-user SAR environment.