Augmented Reality (AR) systems that overlay digital instructions onto the physical world can enhance user performance in complex industrial tasks. Procedural instructions can often be derived from existing approved technical documentation, as reuse reduces authoring effort and ensures compliance with established standards. This paper examines the potential of incorporating a Large Language Model (LLM) to add an intelligent, conversational AR assistant to a static AR manual. We conducted an A/B user study (N = 36) evaluating the effects of adding a conversational assistance layer for a hands-on task on a juice mixer laboratory installation using a HoloLens 2. The baseline condition provided a PDF manual requiring manual step navigation, complemented by situated visual anchors. The second variant supplemented this interface with a conversational assistant that could interpret user queries, provide direct verbal guidance, and automatically jump to the relevant page in the manual. We evaluated these interfaces across three distinct scenarios: a linear task, a non-linear task requiring users to jump between pages for troubleshooting, and a task-handoff where users had to identify the current state of a partially completed procedure. Our findings indicate that despite longer total task completion times, participants using the AI assistant spent significantly less time actively working in the linear and non-linear scenarios, indicating improved task efficiency beyond system latency. Eye-tracking analysis supports the efficiency gain observation as the conversational interface allowed users to focus their visual attention on actively understanding and learning the procedural task. This study highlights the potential of LLM-powered agents for AR guidance, suggesting that overcoming system latency is the critical barrier to their practical deployment in industrial fields.

As virtual reality (VR) headsets become more comfortable and accessible, their growing use in high-stakes, time-critical settings raises concerns about fatigue. Fatigue impairs perceptual and cognitive functioning. It reduces oculomotor accuracy and visual focus, and can lead to an early decline in depth estimation performance. By augmenting the visual presentation with depth information cues, adaptive designs can reduce fatigue-related depth perception errors, enhancing safety and task effectiveness. Explicit cues present depth information directly through text or color, while subtle cues adjust scene properties, such as depth-dependent blur, to convey depth information implicitly without drawing overt attention. We examined how fatigue interacts with different cues in a 27-hour within-subject protocol. Across six overnight sessions (20:00– 07:00), twenty-three participants completed a VR depth perception task at varying fatigue levels and under four cue conditions: no cue (baseline), text, color, and blur. Over the night, vigilance declined, sleepiness and mental effort increased, and simulator sickness rose before stabilizing, independent of cue condition. All augmented cues reduced depth estimation error relative to baseline. Text yielded the largest and most consistent accuracy gains, especially for farther targets and later sessions. At peak fatigue, response probability dipped for text but remained stable for blur and color, indicating an accuracy versus responsiveness trade-off. These results support mixed adaptive designs that default to subtle cues to preserve responsiveness at low alertness and introduce explicit overlays when precise metric information is needed.

Navigation is a fundamental task supporting guided exploration and wayfinding as standalone or as part of other immersive applications. Previous research showed that navigational cues do not only impact wayfinding performance but can also affect perceptual and cognitive processes, e.g. divide attention and impair spatial memory. This study investigates whether varying the density of visual navigation cues can influence situational awareness. Additionally, we examine how cue density affects navigation usability and task performance. We compare three visual cue designs, ranging from high to low density: (1) PathLine, (2) ArrowTrail and (3) TurnMarker. We designed a user study augmenting a 3D scanned digital twin of a building with virtual machinery to simulate a factory floor maintenance task, where the cues guided participants to the next point of interest. A secondary task, the reporting of anomalies, was installed to assess situational awareness. Results showed that performance metrics remained unaffected by cue type, situational awareness and user experience results showed significant differences. Notably, the ArrowTrail cue, the medium-density design, was preferred by most participants and yielded the best overall results, e.g. in terms of anomaly detection and reaction time. These findings suggest that moderate cue density may offer an optimal balance between effective guidance and maintaining environmental awareness.

This paper presents a comprehensive study of virtual 3D object manipulation along 4DoF on real surfaces in mixed reality (MR), using hand-based and tangible interactions. A custom cylindrical tangible proxy leverages affordances of physical knobs and tabletop support for stable input. We evaluate both modalities across isolated tasks (2DoF translation, 1DoF rotation/scaling), semi-combined (3DoF translation+rotation), and full 4DoF compound manipulation. We offer analyses of hand interactions, tangible interactions, and their comparison in MR tasks. For hand interactions, compound tasks required repetitive corrections, increasing completion times—yet surprisingly, rotation errors were smaller in compound tasks than in rotation-only tasks. Tangible interactions exhibited significantly larger errors in translation, rotation, and scaling during compound tasks compared to isolated tasks. Crucially, tangible interactions outperformed hand interactions in precision, likely due to tabletop support and constrained 4DoF design. These findings inform designers opting for hand-only interaction (highlighting trade-offs in compound tasks) and those leveraging tangibles (emphasizing precision gains despite compound-task challenges).

Many industrial sectors rely on well-trained employees that are able to operate complex machinery. In this work, we demonstrate an immersive assistance system powered by Artificial Intelligence (AI) that supports users in performing complex tasks in industrial environments. Our system leverages a Virtual Reality (VR) environment that resembles a juice mixer setup. This digital twin of a physical setup simulates complex industrial machinery used to mix preparations or liquids (e.g., similar to the pharmaceutical industry) and includes various containers, sensors, pumps, and flow controllers. This setup demonstrates our system’s capabilities in a controlled environment while acting as a proof-of-concept for broader industrial applications. The core components of our multimodal AI assistant are a large language model and a speech-to-text model that process a video and audio recording of an expert performing the task in a VR environment. The video and speech input extracted from the expert’s video enables it to provide step-by-step guidance to support users in executing complex tasks. This demonstration showcases the potential of our AI-powered assistant to reduce cognitive load, increase productivity, and enhance safety in industrial environments.

Chronic pain is a major health problem that requires the development of novel treatment strategies to address this growing issue. This paper introduces the application of virtual reality technology for managing chronic pain in outpatient settings. Virtual reality is unique in its ability to engage users through a multisensory experience merging visual, auditory, and sometimes tangible stimuli. We conducted a study involving patients experiencing chronic lumbar and cervical pain to assess its effectiveness in reducing chronic pain. Participants were asked to report their pain scores before, during, and immediately after the virtual reality session. Additionally, the study automatically recorded each participant’s completion time and error rate. These metrics were then analyzed to investigate the potential relationship between pain intensity and task performance. The results demonstrated a significant decrease in pain ratings both during and after the virtual reality session. The findings also revealed an interaction between pain intensity levels and the time spent playing the virtual reality game. Specifically, higher pain levels were associated with shorter completion times for tasks. Additionally, as pain increased, patients’ accuracy in performing tasks decreased. The study we conducted showed a high potential for enhancing distraction levels for patients with chronic pain using virtual reality.

Understanding an environment relies on human sensory systems, with visual perception as the primary source for defining spatial relationships and estimating distances. The visual system uses natural cues, each offering partial information that can lead to bias and conflicts, especially when ambiguous. Virtual Reality (VR) environments challenge these natural depth cues with discrepancies in perspective and variations in light and shadow depiction, leading to potential confusion in depth perception. However, VR also allows for the isolation and study of these cues. This paper introduces artificial subtle cues (texture blur) to enhance natural depth information in VR. Our results show that augmenting natural depth cues with artificial ones improves depth prediction accuracy and spatial relationship awareness. Subtle blur cues enhance depth estimation without participants’ subjective awareness of the augmentation, suggesting that such subtle cueing can effectively enhance depth perception.

Error perception is known to elicit distinct brain patterns, which can be used to improve the usability of systems facilitating human-computer interactions, such as brain-computer interfaces. This requires a high-accuracy detection of erroneous events, e.g., misinterpretations of the user’s intention by the interface, to allow for suitable reactions of the system. In this work, we concentrate on steering-based navigation tasks. We present a combined electroencephalography-virtual reality (VR) study investigating different approaches for error detection and simultaneously exploring the corrective human behavior to erroneous events in a VR flight simulation. We could classify different errors allowing us to analyze neural signatures of unexpected changes in the VR. Moreover, the presented models could detect errors faster than participants naturally responded to them. This work could contribute to developing adaptive VR applications that exclusively rely on the user’s physiological information.

This paper presents the results of two cognitive load studies comparing three augmented reality display technologies: spatial augmented reality, the optical see-through Microsoft HoloLens, and the video see-through Samsung Gear VR. In particular, the two experiments focused on isolating the cognitive load cost of receiving instructions for a button-pressing procedural task. The studies employed a self-assessment cognitive load methodology, as well as an additional dual-task cognitive load methodology. The results showed that spatial augmented reality led to increased performance and reduced cognitive load. Additionally, it was discovered that a limited field of view can introduce increased cognitive load requirements. The findings suggest that some of the inherent restrictions of head-mounted displays materialize as increased user cognitive load.

We draw from the domains of Visual Analytics and Augmented Reality to support a new form of in-situ interactive visual analysis. We present a Situated Analytics model, a novel interaction, and a visualization concept for reasoning support. Situated Analytics has four primary elements: situated information, abstract information, augmented reality interaction, and analytical interaction.

In the last decade, Augmented Reality has become more mature and is widely adopted on mobile devices. Exploring the available information of a user's environment is one of the key applications. However, current mobile Augmented Reality interfaces are very limited compared to the recently emerging big data exploration tools for desktop computers. Our vision is to bring powerful Visual Analytic tools to mobile Augmented Reality.