Extended Reality (XR) headsets enable large, reconfigurable multi-display workspaces and support view manipulation, allowing the workspace to reposition itself around the user. Cursor warping similarly reduces traversal distance and pointer search by reinitialising the cursor at defined locations. Yet when both mechanisms operate together, the spatial relationship between user, displays, and cursor becomes dynamic, and it remains unclear how cursor repositioning behaves when the workspace itself moves. In a study (N=20) of five cursor-warping strategies with two view manipulations, we show that the benefits of both do not automatically combine: workspace motion can disrupt spatial consistency and alter both performance and movement costs. We show that continuous cursor movement in world space is limited compared to alternative warping techniques, and cursor behaviour and view control are tightly coupled. Hence, cursor initialisation and view manipulation must be co-designed to support efficient and comfortable interaction in XR multi-display environments.

Multimodal interaction has long promised to make interfaces more intuitive and effective by combining complementary inputs. Among these, gaze and speech form a compelling pairing: gaze provides rapid spatial grounding, while speech conveys rich semantic information. Together, they offer rich cues for understanding user behaviour and intent. Yet despite decades of exploration, the research remains fragmented, making this synthesis timely as these inputs mature and are integrated into consumer-ready devices. This scoping review examined 103 studies published between 1991 and 2025, organised into explicit, where users intentionally provide gaze and speech, and implicit, where systems leverage users' natural behaviours to support interaction. Across both, we identified recurring ways for combining gaze and speech to resolve ambiguity, ground references, and support adaptivity. We contribute a synthesis of research on their combined use while highlighting challenges of temporal alignment, fusion and privacy, offering guidance for future research toward richer multimodal human-computer interaction.

Visual alignment tasks, such as perspective pointing, inherently involve ambiguity because either the left or right eye can dominate, serving as the vantage point. Previous work presented eye dominance as dynamic, influenced by horizontal gaze angle to the target. In this work, we investigate prediction based on both target-based and user-dependent features, with machine learning models trained from data collected from a perspective-pointing task (N=28). We confirm that vertical target angle influences eye dominance, strengthening the effect of horizontal angles, specifically below eye level. Models using target information alone performed poorly, particularly for left-eye cases. Including user-specific features improved accuracy and class balance, with gradient boosted classifiers achieving the highest performance. These results underscore the significance of personalized features for eye dominance prediction, with practical implications for virtual reality (VR) interfaces.

Older adults, a growing demographic, face an increased risk of experiencing loneliness and are less exposed to emerging communication technologies. Augmented reality (AR) avatars and telepresence robots have been proposed as tools to foster social connection, yet their suitability for older users remains underexplored. We present an exploratory study with ten healthy older adults who engaged in both conversational and spatial collaboration tasks using AR avatar-mediated communication, robot-mediated communication, and face-to-face interaction. We collected self-reported measures of co-presence, social presence, closeness, uncanny valley, preferences, and open feedback. Our findings suggest that telepresence robots enhanced co-presence, while avatars were valued for their expressivity and humanlike qualities. Task type influenced co-presence in spatial collaboration only during communication using the telepresence robot. Other measures, such as social presence and closeness, were unaffected by task type or representation. While neither technology outperformed face-to-face interaction, both were positively received, underscoring their potential to address the social needs of older adults and highlighting the importance of enhancing nonverbal expressivity, particularly nonverbal cues in mediated communication. Ultimately, our results contribute to the fundamental understanding of mediated communication with older adults, motivating further empirical work to confirm and extend these findings.

Figure 1: The experimental setup for investigating eye dominance in virtual reality (VR).A participant, wearing a VR headset, aligns a virtual cursor with a distant target using a controller.The two insets display the rendering for the left and right eye views, demonstrating target-controller alignment with the right eye only.In this trial, the participant is right-eye dominant.

A replication attempts to confirm outcomes of earlier research, which is critical in validating and generalizing scientific findings. Yet, its prevalence and practices remain underexplored in Augmented and Virtual Reality (AR/VR) research. To address this, we present a scoping literature review of replication studies within IEEE ISMAR and IEEE VR, spanning 15 years from 2010 to 2024. Our analysis revealed that replication in AR/VR research is rare. Of 2167 total papers reviewed, less than 4 % were identified as replication studies. Of these, conceptual replications were the predominant type (57 %). Most of those were studies in VR (67 %) and within-subject designs (66 %). Complementing the literature survey results, we conducted an online survey with 61 participants about their experiences with replication studies and found that 39 % of them had conducted a replication study. However, limited resources and external motivation hamper the execution of replication studies among these AR/VR researchers. Combining the findings from our literature review and online survey, we discuss the current state of replication research and the factors contributing to its infrequency. We provide recommendations to improve AR/VR research replication practices, focusing on research culture and reporting, and discuss ongoing challenges.

Gaze is fast and intuitive for raycasting, but lacks a way to control depth for input in 3D. We propose HeadDepth to augment gaze with vertical head rotation (pitch) to control depth along the line of sight, and investigate three pitch-to-depth mappings: Relative maps pitch velocity to changes in depth; EdgeGain adds dynamic gain dependent at eccentric gaze angles; and PingPong provides an absolute back-and-forth mapping that repeats at different pitch angles. HeadDepth is not as fast as controller-based RayCursor but has the advantage of being hands-free. In a user study, all three variants proved effective for 3D positioning; PingPong required least effort and EdgeGain was least affected by differences in tasks. Eye-head coordination and task had a significant effect, as head movements in support of gaze can affect depth control synergistically or antagonistically. Our results have significance beyond HeadDepth as they generalize to any interaction where head rotational input concurs with gaze fixation on visual feedback. Interaction designs may benefit from dynamic adjustment to eye-in-head angle to prevent discomfort and maintain objects within the user's field of view.

Avatars and telepresence robots have long received attention for remote communication. However, the specific nature of their physicality, expressiveness, and mobility may affect their usefulness for different tasks. This work compares using an avatar (presented in augmented reality) and a telepresence robot to Face-to-Face (F2F) communication during different communication tasks: free conversation, negotiation, and referential communication with movement. We conducted a user study (split-plot design, N=54) with the type of representation of the conversational partner as the within variable and the communication task as the between variable. Our results show that the type of task, especially referential communication with movement, influenced the perceived attention to nonverbal cues and closeness. Generally, gestures and body movements received the least focus with telepresence robots. Gestures in avatars and F2F drew similar attention, which we attribute to the avatar's tracking fidelity. Gaze received less attention in both avatar- and robot-mediated communication compared to F2F, while facial expressions on the robot's screen heightened attention compared to avatars. These findings advance the fundamental understanding of mediated communication and support researchers and practitioners in shaping the design of communication applications beyond today's video calls.

Head pointing is widely used for hands-free input in head-mounted displays (HMDs). The primary role of head movement in an HMD is to control the viewport based on absolute mapping of head rotation to the 3D environment. Head pointing is conventionally supported by the same 1:1 mapping of input with a cursor fixed in the centre of the view, but this requires exaggerated head movement and limits input granularity. In this work, we propose to adopt dynamic gain to improve ergonomics and precision, and introduce the HeadShift technique. The design of HeadShift is grounded in natural eye-head coordination to manage control of the viewport and the cursor at different speeds. We evaluated HeadShift in a Fitts’ Law experiment and on three different applications in VR, finding the technique to reduce error rate and effort. The findings are significant as they show that gain can be adopted effectively for head pointing while ensuring that the cursor is maintained within a comfortable eye-in-head viewing range.

Abstract Behavioral realism plays a crucial role in virtual and augmented reality (VR/AR). Various avatar animation techniques, ranging from full-body motion capture to single-point inverse kinematics (IK), offer different levels of realism. While the animation of a user’s own avatar influences embodiment, the perceived realism of others’ avatars is equally important for immersion. This study ( N = 53) examines how users in smartphone AR, head-mounted display (HMD) AR, and VR perceive the behavioral realism of avatars animated with single-point IK compared to those driven by full-body motion capture. In addition, we explore whether the congruence between visual fidelity of an avatar and tracking accuracy affects perception. Our findings indicate that full-body motion capture produces significantly higher perceived realism than single-point IK, but the type of device does not have measurable impact. Furthermore, while congruence between visual realism and tracking fidelity was expected to play a role, our results suggest that its influence is limited. Despite lower realism than motion capture, modern IK techniques are still perceived positively, highlighting their viability for multi-user AR and VR applications.

Head-mounted displays let users explore virtual environments through a viewport that is coupled with head movement. In this work, we investigate gaze as an alternative modality for viewport control, enabling exploration of virtual worlds with less head movement. We designed three techniques that leverage gaze based on different eye movements: Dwell Snap for viewport rotation in discrete steps, Gaze Gain for amplified viewport rotation based on gaze angle, and Gaze Pursuit for central viewport alignment of gaze targets. All three techniques enable 360-degree viewport control through naturally coordinated eye and head movement. We evaluated the techniques in comparison with controller snap and head amplification baselines, for both coarse and precise viewport control, and found them to be as fast and accurate. We observed a high variance in performance which may be attributable to the different degrees to which humans tend to support gaze shifts with head movement.

Behavioral realism is a key element in social virtual and augmented reality (VR/AR). Different techniques, from full-body to single-point IK, are used to animate virtual humans, providing varying levels of behavioral realism. While the animation method of the user’s own avatar is important, e.g., considering embodiment, the animation of others is equally important to display a convincing immersive experience. This study ($\mathrm{N}=36$) investigates how users (in AR or VR) rate the behavioral realism of others when the other person’s avatar is animated using a single point Inverse Kinematics (IK) and compares results to a full-body motion-capture-driven avatar. In addition, we investigate whether congruence between visual realism and tracking quality matters. Our results reveal that animations based on full-body motion capture are found to result in higher perceived behavioral realism compared to those using single-point inverse kinematics (IK), that device type has no influence, and that congruence seems to matter less. Despite not reaching the same level of realism as motion-captured animations, our results suggest the applicability of state-of-the-art IK techniques for multi-user applications in AR and VR as, in general, participants rated both positively.

Augmented Reality (AR) and Virtual Reality (VR) are essential tools for researchers and practitioners, serving purposes from training to entertainment: many of these applications rely on agents. This study explores the impact of agent characteristics on user reactions, focusing on gaze as a primary visual attention indicator in AR and VR. While existing research has investigated the agent’s gaze and its influence on the user, it is unclear how the agent’s auralization and visualization influence gaze behaviour. We investigate this by studying the impact of rendering style and type of audio on gaze behaviour during a narrative AR experience. Participants listened to a story with the agent visualized as a cartoon-style or realistic virtual human and auralized with spatial or non-spatial audio. The results revealed that the agent’s rendering style significantly influenced gaze behaviour, with cartoon-style agents capturing more visual attention. Audio variations did not yield significant differences. Together, our findings inform the design of AR user interfaces with agents, suggesting that low-realism visualizations are more captivating and, thus, more suitable for experiences where the user is supposed to look at the storyteller.

The appearance of virtual humans (avatars and agents) has been widely explored in immersive environments. However, virtual humans’ movements and associated sounds in real-world interactions, particularly in Augmented Reality (AR), are yet to be explored. In this paper, we investigate the influence of three distinct movement patterns (circle, side-to-side, and standing), two rendering styles (realistic and cartoon), and two types of audio (spatial audio and non-spatial audio) on emotional responses, social presence, appearance and behavior plausibility, audiovisual coherence, and auditory plausibility. To enable that, we conducted a study (N=36) where participants observed an agent reciting a short fictional story. Our results indicate an effect of the rendering style and the type of movement on the subjective perception of the agents behaving in an AR environment. Participants reported higher levels of excitement when they observed the realistic agent moving in a circle compared to the cartoon agent or the other two movement patterns. Moreover, we found an influence of agent’s movement pattern on social presence and higher appearance and behavior plausibility for the realistic rendering style. Regarding audiovisual spatial coherence, we found an influence of rendering style and type of audio only for the cartoon agent. Additionally, the spatial audio was perceived as more plausible than non-spatial audio. Our findings suggest that aligning realistic rendering styles with realistic auditory experiences may not be necessary for 1-1 listening experiences with moving sources. However, movement patterns of agents influence excitement and social presence in passive unidirectional communication scenarios.

Augmented reality (AR) presents vivid opportunities for interpersonal communication. With the growing diversity of social AR users, understanding their unique needs and perceptions becomes crucial. This study delves into how younger, middle-aged, and older adults perceive avatars with different aging attributes and degrees of realism, focusing on their anticipated user experience within a social AR system. We conducted an online within-subjects experiment involving $N = 2086$ age-diverse participants from Germany who assessed a set of nine gender-matched avatars for their perceived social attractiveness (research question 1 = RQ1) and the likelihood of selecting these avatars for self-representation in social AR (RQ2). The evaluated avatars represented different age groups (younger, middle-aged, and older) and levels of realism (low, medium, and high). We validated both the created avatars and our experimental setup and employed a linear mixed-effects modeling approach to analyze the data. Our findings unveiled a strong preference for younger high-realism avatars as communication partners (RQ1), which was consistent across all participant age groups. Similarly, participants favored younger high-realism avatars for self-representation in social AR (RQ2). However, older adults were more inclined to opt for avatars resembling their actual age. The study highlights the prevalence of age-related stereotypes in avatar-based communication. Similar to face-to-face social interactions, these stereotypes tend to render older avatars less socially attractive than their younger counterparts, irrespective of the avatar’s degree of realism. Our results invite considerations on how to combat these stereotypes through a more thoughtful and inclusive avatar design process that encompasses a broader spectrum of aging attributes.

Previous studies have explored the perception of various types of embodied avatars in immersive environments. However, the impact of eye and face tracking with personalized avatars is yet to be explored. In this paper, we investigate the impact of eye and face tracking on embodiment, enfacement, and the uncanny valley with four types of avatars using a VR-based mirroring task. We conducted a study (N=12) and created self-avatars with two rendering styles: a cartoon avatar (created in an avatar generator using a picture of the user’s face) and a photorealistic scanned avatar (created using a 3D scanner), each with and without eye and face tracking and respective adaptation of the mirror image. Our results indicate that adding eye and face tracking can be beneficial for certain enfacement scales (belonged), and we confirm that compared to a cartoon avatar, a scanned realistic avatar results in higher body ownership and increased enfacement (own face, belonging, mirror) — regardless of eye and face tracking. We critically discuss our experiences and outline the limitations of the applied hardware and software with respect to the provided level of control and the applicability for complex tasks such as displaying emotions. We synthesize these findings into a discussion about potential improvements for facial animation in VR and highlight the need for a better level of control, the integration of additional sensing and processing technologies, and an objective metric for comparing facial animation systems.

The combination of smartphone Augmented Reality (AR) and Virtual Reality (VR) makes it possible for on-site and remote users to simultaneously explore a physical space and its digital twin through an asymmetric Collaborative Virtual Environment (CVE). In this paper, we investigate two spatial awareness visualizations to enable joint exploration of a space for dyads consisting of a smartphone AR user and a head-mounted display VR user. Our study revealed that both, a mini-map-based method and an egocentric compass method with a path visualization, enabled the on-site visitors to locate and follow a virtual companion reliably and quickly. Furthermore, the embodiment of the AR user by an inverse kinematics avatar allowed the use of natural gestures such as pointing and waving which was preferred over text messages by the participants of our study. In an expert review in a museum and its digital twin we observed an overall high social presence for on-site AR and remote VR visitors and found that the visualizations and the avatar embodiment successfully facilitated their communication and collaboration.

Augmented Reality (AR) and Virtual Reality (VR) are pushing from the labs towards consumers, especially with social applications. These applications require visual representations of humans and intelligent entities. However, displaying and animating photo-realistic models comes with a high technical cost while low-fidelity representations may evoke eeriness and overall could degrade an experience. Thus, it is important to carefully select what kind of avatar to display. This article investigates the effects of rendering style and visible body parts in AR and VR by adopting a systematic literature review. We analyzed 72 papers that compare various avatar representations. Our analysis includes an outline of the research published between 2015 and 2022 on the topic of avatars and agents in AR and VR displayed using head-mounted displays, covering aspects like visible body parts (e.g., hands only, hands and head, full-body) and rendering style (e.g., abstract, cartoon, realistic); an overview of collected objective and subjective measures (e.g., task performance, presence, user experience, body ownership); and a classification of tasks where avatars and agents were used into task domains (physical activity, hand interaction, communication, game-like scenarios, and education/training). We discuss and synthesize our results within the context of today's AR and VR ecosystem, provide guidelines for practitioners, and finally identify and present promising research opportunities to encourage future research of avatars and agents in AR/VR environments.

Integrating taste in AR/VR applications has various promising use cases - from social eating to the treatment of disorders. Despite many successful AR/VR applications that alter the taste of beverages and food, the relationship between olfaction, gustation, and vision during the process of multisensory integration (MSI) has not been fully explored yet. Thus, we present the results of a study in which participants were confronted with congruent and incongruent visual and olfactory stimuli while eating a tasteless food product in VR. We were interested (1) if participants integrate bi-modal congruent stimuli and (2) if vision guides MSI during congruent/incongruent conditions. Our results contain three main findings: First, and surprisingly, participants were not always able to detect congruent visual-olfactory stimuli when eating a portion of tasteless food. Second, when confronted with tri-modal incongruent cues, a majority of participants did not rely on any of the presented cues when forced to identify what they eat; this includes vision which has previously been shown to dominate MSI. Third, although research has shown that basic taste qualities like sweetness, saltiness, or sourness can be influenced by congruent cues, doing so with more complex flavors (e.g., zucchini or carrot) proved to be harder to achieve. We discuss our results in the context of multimodal integration, and within the domain of multisensory AR/VR. Our results are a necessary building block for future human-food interaction in XR that relies on smell, taste, and vision and are foundational for applied applications such as affective AR/VR.

Smartphone Augmented Reality (AR) has already provided us with a plethora of social applications such as Pokemon Go or Harry Potter Wizards Unite. However, to enable smartphone AR for social applications similar to VRChat or AltspaceVR, proper user tracking is necessary to accurately animate the avatars. In Virtual Reality (VR), avatar tracking is rather easy due to the availability of hand-tracking, controllers, and HMD whereas smartphone AR has only the back-(and front) camera and IMUs available for this task. In this paper we propose ARIKA, a tracking solution for avatars in smartphone AR. ARIKA uses tracking information from ARCore to track the users hand position and to calculate a pose using Inverse Kinematics (IK). We compare the accuracy of our system against a commercial motion tracking system and compare both systems with respect to sense of agency, self-location, and body-ownership. For this, 20 participants observed their avatars in an augmented virtual mirror and executed a navigation and a pointing task. Our results show that participants felt a higher sense of agency and self location when using the full body tracked avatar as opposed to IK avatars. Interestingly and in favor of ARIKA, there were no significant differences in body-ownership between our solution and the full-body tracked avatars. Thus, ARIKA and it’s single-camera approach is valid solution for smartphone AR applications where body-ownership is essential.

Abstract When operating a conditionally automated vehicle, humans occasionally have to take over control. If the driver is out of the loop, a certain amount of time is necessary to gain situation awareness. This work evaluates the potential of stereoscopic 3D (S3D) dashboards for presenting smart S3D take-over-requests (TORs) to support situation assessment. In a driving simulator study with a 4 × 2 between-within design, we presented 3 smart TORs showing the current traffic situation and a baseline TOR in 2D and S3D to 52 participants doing the n-back task. We further investigate if non-standard locations affect the results. Take-over performance indicates that participants looked at and processed the TORs’ visual information and by that, could perform more safe take-overs. S3D warnings in general, as well as warnings appearing at the participants’ focus of attention and warnings at the instrument cluster, performed best. We conclude that visual warnings, presented on an S3D dashboard, can be a valid option to support take-over while not increasing workload. We further discuss participants’ gaze behavior in the context of visual warnings for automotive user interfaces.

Semantic segmentation aims at providing a fine-grained image prediction by assigning each pixel to a specific semantic category. Convolutional neural networks offer significant benefits for solving this problem. However, the success of such networks is closely related to the availability of corresponding data sets. To facilitate semantic segmentation in a broader range of scenarios, such as augmented reality in outdoor environments or universal image-to-image translation, adequate training data sets are necessary. We present OUTSIDE15k, a large-scale data set for semantic segmentation of universal outdoor scenes. The data is labeled with 24 different semantic classes. The images contain multiple outdoor scenarios and cover a variety of different resolutions. Additionally, we present OUTSIDE-Net, an improved neural network architecture integrating multi-level pooling, feature fusion, and a spatial mask for semantic segmentation of universal outdoor scenes. It extracts spatial and semantic features from the input images to perform the segmentation. With the presented data set, we show the capability of our network which outperforms state-of-the-art approaches by achieving up to 91.5% pixel accuracy.

Recent developments in augmented reality for TV productions encouraged broadcasters to enhance interaction with virtual content for moderators. However, traditional interaction methods are considered distracting and not intuitive. To overcome these issues, we performed a gesture elicitation study with a follow-up evaluation. For this, we considered TV moderators as primary users of the gestures as well as viewers as recipients. The elicited gesture set consists of five gestures for two types of camera shots (long shot and close shot). Findings of the evaluation study indicate that the derived set of gestures requires low physical and concentration effort from moderators. Also, both moderators and viewers found them appropriate to be used in TV with respect to understandability, distraction, likeability, and appropriateness. Using these gestures would allow moderators to control AR content in TV and tell stories in a modern and more expressive way.

Many virtual reality (VR) applications rely on passive haptics where virtual objects have a real counterpart that provides tactile feedback. In addition to that, many VR applications do not provide accurate hand representations, especially when there is a high chance of occlusion as this makes vision-based tracking problematic. Hence, we investigated how a simple hand representation affects user experience and presence when interacting with passive haptic controls in a virtual environment. We report on a between-subject user study where N = 45 participants experienced one of three conditions (no hands at all, hands represented as a rigid 3D model, and hands represented as a rigid 3D model with a snapping mechanism). Our results indicate that a simple hand representation using a 3D model of hands paired with a snapping mechanism significantly increases presence and user experience. That indicates that this simple and low-cost technique is effective to improve the VE as a whole. This, in return, provides a chance for improvement for many VR applications with passive haptics.

In recent years, stereoscopic 3D (S3D) displays have shown promising results on user experience, for navigation, and critical warnings when applied in cars. However, previous studies have only investigated these displays in non-interactive use-cases. So far, interacting with stereoscopic 3D content in cars has not been studied. Hence, we investigated how people interact with large S3D dashboards in automated vehicles (SAE level 4). In a user-elicitation study (N=23), we asked participants to propose interaction techniques for 24 referents while sitting in a driving simulator. Based on video recordings and motion tracking data of 1104 proposed interactions containing gestures and other input modalities, we grouped the gestures per task. Overall, we can report a chance-corrected agreement rate of k = 0.232 and by that, a medium agreement among participants. Based on the agreement rates, we defined two sets of gestures: a basic and a holistic version. Our results show that participants intuitively interact with S3D dashboards and that they prefer mid-air gestures that either directly manipulate the virtual object or operate on a proxy object. We further compare our results with similar results in different settings and provide insights on factors that have shaped our gesture set.

Although visual attention cues are of particular importance for driving simulation tasks, research on the relationship of visual attention and simulator sickness is scarce. This exploratory study is aimed at investigating this relation with a laboratory study in a fixed-based driving simulator (N = 36). No correlation between visual attention and simulator sickness was shown, but the direction of the relation shows a negative tendency.

In recent years, virtual reality products have found their way into our life. They have changed our entertainment experience in many ways, including the way people consume adult entertainment. With the immersion and presence offered by VR technologies, a new level of user experience in adult entertainment might be possible. To explore these possibilities, we conducted a qualitative user study with 12 participants to investigate participants views on realism and interactivity in immersive adult content videos and a virtual reality erotic digital games. Results indicate that a high sense interactivity is no less important than a high level of visual realism.

Up to now, most driving simulators use either small monitors or large immersive projection setups like 2D/3D screens or a CAVE. The recent improvements of VR-HMDs led to an increased application in driving simulation. However, the influence and comparability of various VR and non-VR displays has been hardly investigated. We present results of a user study investigating the different influence of non-VR (2D, stereoscopic 3D) and VR (HMD) on physiological responses, simulation sickness, and driving performance within a single driving simulator. In the study, 94 participants performed the Lane Change Task. Results indicate that a VR-HMD leads to similar data as stereoscopic 3D or 2D screens. We observed no significant difference regarding physiological responses or lane change performance. However, we measured significantly increased simulator sickness in the VR-HMD condition compared to stereoscopic 3D.