Older adults, a growing demographic, face an increased risk of experiencing loneliness and are less exposed to emerging communication technologies. Augmented reality (AR) avatars and telepresence robots have been proposed as tools to foster social connection, yet their suitability for older users remains underexplored. We present an exploratory study with ten healthy older adults who engaged in both conversational and spatial collaboration tasks using AR avatar-mediated communication, robot-mediated communication, and face-to-face interaction. We collected self-reported measures of co-presence, social presence, closeness, uncanny valley, preferences, and open feedback. Our findings suggest that telepresence robots enhanced co-presence, while avatars were valued for their expressivity and humanlike qualities. Task type influenced co-presence in spatial collaboration only during communication using the telepresence robot. Other measures, such as social presence and closeness, were unaffected by task type or representation. While neither technology outperformed face-to-face interaction, both were positively received, underscoring their potential to address the social needs of older adults and highlighting the importance of enhancing nonverbal expressivity, particularly nonverbal cues in mediated communication. Ultimately, our results contribute to the fundamental understanding of mediated communication with older adults, motivating further empirical work to confirm and extend these findings.

In Mixed Reality (MR), in-the-moment guidance can support physical tasks under time and attention constraints. In contexts where such guidance is embodied, users must rapidly evaluate whether to trust and follow an advisor’s recommendations. Prior work has examined how avatar representation shapes trust and social perception, but less is known about user responses when guidance conflicts and multiple embodied agents compete for attention in real time. We study trust-relevant perception and behavior in an MR paradigm where two co-present agents represent the same advisor and deliver step-by-step instructions with predefined disagreements. In a within-subjects study, we compared a volumetric avatar (VA) and a high-fidelity 3D character avatar (CA), holding identity and expressive behavior constant, and measured trust, social presence, and behavioral adherence during contradictory steps. Rendering style produced no detectable differences in trust, social presence, or co-presence. Instead, participants managed disagreement through mixed adherence and switching between agents, suggesting that instruction competition and time pressure strongly shape decision-making. These findings position conflicting guidance as a distinct context for studying trust in embodied agents, where compliance may reflect rapid prioritization rather than stable commitment to a representation. We discuss implications for designing and evaluating MR agents when timing and attention constraints can overshadow representation cues.

Smartphone Augmented Reality (AR) has already introduced numerous social applications. However, extending smartphone AR to support social experiences akin to VRChat or AltspaceVR requires robust user tracking to animate avatars accurately. While Virtual Reality (VR) benefits from devices like hand-tracking systems, controllers, and Head-mounted displays (HMDs) for precise avatar tracking, smartphone AR is limited to back and front cameras alongside inertial measurement units (IMUs). Previous work has explored animating smartphone users' avatars using singlepoint Inverse Kinematics (IK) derived from smartphone tracking data. In this paper, we propose an enhanced approach that combines face tracking with single-point IK to improve avatar animation. Our system leverages ARKit to track hand and face positions and orientations to estimate user poses through IK. We evaluate our system's performance against a commercial motion tracking system and traditional single-point IK in terms of tracking accuracy, user embodiment, and user experience. Our findings suggests that our dual-point IK (DPIK) method demonstrated significantly lower mean tracking errors and more consistent performance across joints compared to traditional single-point IK. Although statistical differences in embodiment measures were not significant, users reported a stronger sense of control and agency, particularly in head-tracked movements. Participants also favored DPIK in terms of both Pragmatic and Hedonic qualities, highlighting its potential for improving avatar representation in social AR experiences.

Video pass-through (VPT) techniques often cause visual distortions like the ‘telescope eye' effect, which leads to less immersion and spatial awareness in Extended Reality (XR). This study investigates the potential of 3D Gaussian Splatting (3DGS) as a rendering alternative to video pass-through, particularly focusing on its ability to mitigate common perceptual artifacts and enhance immersion. To evaluate the efficacy of 3DGS in an XR context, we conducted a user study comparing two visualization types: (1) conventional VPT and (2) 3DGS-based scene representation. Three distinct scenes with high detail, natural depth perception, and reflective surfaces were included in the study. An empirical study was conducted with a structured questionnaire that inquired about the level of immersion and the impact of the visual stimuli. Our results provide insights into the scalability and hardware considerations for implementing 3DGS in realtime interactive environments. The evaluation results indicated that users with a 3DGS visualization experienced a reduction in visual distortion and an enhancement in the perceived realism of objects. The findings show that 3DGS can serve as an effective alternative to VPT, potentially improving realism and reducing perceptual artifacts in XR applications.

The Audiovisual Multimodal Interaction Suite (AMIS) is an open-source dataset and accompanying Unity-based demo implementation designed to aid research on immersive media communication and social XR environments. AMIS features synchronized audiovisual recordings of three actors performing monologues and participating in dyadic conversations across four modalities: talking-head videos, full-body videos, volumetric avatars, and personalized animated avatars. These recordings can be used to simulate scenarios such as traditional video conferences or XR meetings with 3D avatars in controlled and replicable environments. The limitations of existing datasets, which include a restricted number of audiovisual formats, a narrow application focus, and suboptimal inclusion of verbal and non-verbal cues, are addressed by AMIS. With AMIS Studio, a Unity-based demonstrator, researchers can explore the recordings and compare the different audiovisual formats in VR scenes. This paper outlines the creation of AMIS, its design considerations, and how it may be applied in interdisciplinary domains, including cognitive psychology, audiovisual quality assessment, and social XR research.

Current and, above all, future mixed reality services and systems collect a large amount of different personal and environmental data of the users and other involved persons. Current legal regulations on data protection and governance do not appear to adequately meet these challenges. This paper looks at data collection in Mixed Reality systems in order to create a greater awareness for its far-reaching privacy-related implications. We present a straightforward scenario where a user named Melanie interacts with both her environment and another user. It is stipulated that Melanie is not a first-time user, and all interactions and environmental factors are predetermined by an existing data profile. This paper examines the technical design of the scenario, the types of data collected, and their legal implications. To address the far-reaching privacy implications and shortcomings in the current legal framework, we propose technical solutions for less privacy-encroaching data uses and recommend the implementation of better legal instruments and safeguards in the future. In conclusion we suggest that better legal instruments and safeguards are needed in future.

Avatars and telepresence robots have long received attention for remote communication. However, the specific nature of their physicality, expressiveness, and mobility may affect their usefulness for different tasks. This work compares using an avatar (presented in augmented reality) and a telepresence robot to Face-to-Face (F2F) communication during different communication tasks: free conversation, negotiation, and referential communication with movement. We conducted a user study (split-plot design, N=54) with the type of representation of the conversational partner as the within variable and the communication task as the between variable. Our results show that the type of task, especially referential communication with movement, influenced the perceived attention to nonverbal cues and closeness. Generally, gestures and body movements received the least focus with telepresence robots. Gestures in avatars and F2F drew similar attention, which we attribute to the avatar's tracking fidelity. Gaze received less attention in both avatar- and robot-mediated communication compared to F2F, while facial expressions on the robot's screen heightened attention compared to avatars. These findings advance the fundamental understanding of mediated communication and support researchers and practitioners in shaping the design of communication applications beyond today's video calls.

Abstract Behavioral realism plays a crucial role in virtual and augmented reality (VR/AR). Various avatar animation techniques, ranging from full-body motion capture to single-point inverse kinematics (IK), offer different levels of realism. While the animation of a user’s own avatar influences embodiment, the perceived realism of others’ avatars is equally important for immersion. This study ( N = 53) examines how users in smartphone AR, head-mounted display (HMD) AR, and VR perceive the behavioral realism of avatars animated with single-point IK compared to those driven by full-body motion capture. In addition, we explore whether the congruence between visual fidelity of an avatar and tracking accuracy affects perception. Our findings indicate that full-body motion capture produces significantly higher perceived realism than single-point IK, but the type of device does not have measurable impact. Furthermore, while congruence between visual realism and tracking fidelity was expected to play a role, our results suggest that its influence is limited. Despite lower realism than motion capture, modern IK techniques are still perceived positively, highlighting their viability for multi-user AR and VR applications.

Abstract To achieve realistic lighting in augmented reality (AR), virtual reality (VR), and similar applications especially with stringent realtime constraints, environmental lighting is often captured and stored in light probes, which are strategically placed throughout the scene. This study explores the optimal positioning of light probes to enhance the realism of extended realities. A novel spatial geometry-based method for light probe placement is proposed and compared against three alternative techniques. They are evaluated across various 3D Gaussian splatting environments with different probe distributions. The techniques are compared to ground truth renderings that are obtained using a pipeline that combines fast Gaussian rasterization with ray tracing. The findings demonstrate that the careful placement of a minimal number of probes can effectively replicate realistic lighting in complex environments. The geometry-based optimization method outperforms the alternatives in indoor environments by providing an efficient realtime solution with fewer probes. This optimized approach is suitable for direct implementation in realtime applications such as AR, VR, game development, and virtual productions, offering both improved lighting realism and optimized computational efficiency.

Behavioral realism is a key element in social virtual and augmented reality (VR/AR). Different techniques, from full-body to single-point IK, are used to animate virtual humans, providing varying levels of behavioral realism. While the animation method of the user’s own avatar is important, e.g., considering embodiment, the animation of others is equally important to display a convincing immersive experience. This study ($\mathrm{N}=36$) investigates how users (in AR or VR) rate the behavioral realism of others when the other person’s avatar is animated using a single point Inverse Kinematics (IK) and compares results to a full-body motion-capture-driven avatar. In addition, we investigate whether congruence between visual realism and tracking quality matters. Our results reveal that animations based on full-body motion capture are found to result in higher perceived behavioral realism compared to those using single-point inverse kinematics (IK), that device type has no influence, and that congruence seems to matter less. Despite not reaching the same level of realism as motion-captured animations, our results suggest the applicability of state-of-the-art IK techniques for multi-user applications in AR and VR as, in general, participants rated both positively.

In order to achieve realistic lighting in augmented reality (AR), virtual reality (VR), and other virtual environments with stringent real-time requirements, environmental lighting is captured and stored in light probes that are strategically positioned throughout these environments. This study investigates the optimal positioning of light probes based on the environment to achieve realistic virtual representations. A novel spatial geometry-based approach for light probe placement is introduced, validated in a comparative analyses against three alternative methods across diverse virtual environments, each utilizing different probe quantities. The investigation reveals that strategic deployment of a minimal number of probes can effectively simulate realistic illumination across complex environmental scenarios. The geometry-based optimization outperforms alternative methods by achieving efficient real-time probe placement with minimal number of probes. This optimized methodology can be directly implemented at runtime in various applications, including AR, VR, game development, and virtual productions, thereby enhancing lighting realism while optimizing computational resources.

Augmented Reality (AR) and Virtual Reality (VR) are essential tools for researchers and practitioners, serving purposes from training to entertainment: many of these applications rely on agents. This study explores the impact of agent characteristics on user reactions, focusing on gaze as a primary visual attention indicator in AR and VR. While existing research has investigated the agent’s gaze and its influence on the user, it is unclear how the agent’s auralization and visualization influence gaze behaviour. We investigate this by studying the impact of rendering style and type of audio on gaze behaviour during a narrative AR experience. Participants listened to a story with the agent visualized as a cartoon-style or realistic virtual human and auralized with spatial or non-spatial audio. The results revealed that the agent’s rendering style significantly influenced gaze behaviour, with cartoon-style agents capturing more visual attention. Audio variations did not yield significant differences. Together, our findings inform the design of AR user interfaces with agents, suggesting that low-realism visualizations are more captivating and, thus, more suitable for experiences where the user is supposed to look at the storyteller.

In unusual environments and situations like extreme sports, underground environments, underwater, conflict zones or outer space, human sensory perception is often limited, which can adversely affect task performance, situational awareness, and the feeling of presence. This paper explores the efficacy of augmenting human perception with audiovisual information to support individuals in such contexts. Specifically, we investigate the impact of audiovisual augmentation on presence, situational awareness, and task performance. To conduct this research, we designed a virtual reality (VR) simulation of a space mission aboard the International Space Station (ISS). Within this simulation, participants were tasked with performing maintenance activities while receiving artificial augmentations. We conducted a user study involving 43 participants who performed the same maintenance task under four augmentation conditions: audio cues, visual cues, audiovisual cues, and no cues. Our findings reveal that adding audiovisual information significantly enhances performance. Participants with audiovisual augmentations had a 60% improvement in task completion speed compared to those without augmentations. Moreover, the workload was substantially reduced, and the sense of presence was increased when audiovisual support was used. The results highlight the potential of audiovisual augmentation as a valuable tool for individuals engaging in situations with reduced audiovisual perception. The insights demonstrating the versatility and promise of audiovisual augmentation in enhancing task performance.

Given the profound impact of staying socially connected on the well-being of older adults, this study explores the potential of augmented reality (AR) systems to enrich their social lives. A wearable AR communication system prototype was developed and tested in a user study involving N = 16 older adults from Germany. Participants wore an AR headset and engaged in a conversation task with a remote person represented by an avatar. Older adults’ experiences were assessed using think-aloud protocols, qualitative observations, posttest questionnaires, and semi-structured oral interviews. Preliminary findings indicate overall participant satisfaction, with minimal observed difficulties in headset usage and avatar-mediated interpersonal communication. The positive engagement during AR conversations highlights the system’s potential to provide positive communication experiences among older individuals. This work-in-progress paper introduces the developed system prototype and outlines the conducted user study. Further data analyses will provide deeper insights into older adults’ experiences with the system. The results will contribute to refining the prototype and offer valuable insights for the development of AR communication systems tailored to the needs and preferences of older adults.

The appearance of virtual humans (avatars and agents) has been widely explored in immersive environments. However, virtual humans’ movements and associated sounds in real-world interactions, particularly in Augmented Reality (AR), are yet to be explored. In this paper, we investigate the influence of three distinct movement patterns (circle, side-to-side, and standing), two rendering styles (realistic and cartoon), and two types of audio (spatial audio and non-spatial audio) on emotional responses, social presence, appearance and behavior plausibility, audiovisual coherence, and auditory plausibility. To enable that, we conducted a study (N=36) where participants observed an agent reciting a short fictional story. Our results indicate an effect of the rendering style and the type of movement on the subjective perception of the agents behaving in an AR environment. Participants reported higher levels of excitement when they observed the realistic agent moving in a circle compared to the cartoon agent or the other two movement patterns. Moreover, we found an influence of agent’s movement pattern on social presence and higher appearance and behavior plausibility for the realistic rendering style. Regarding audiovisual spatial coherence, we found an influence of rendering style and type of audio only for the cartoon agent. Additionally, the spatial audio was perceived as more plausible than non-spatial audio. Our findings suggest that aligning realistic rendering styles with realistic auditory experiences may not be necessary for 1-1 listening experiences with moving sources. However, movement patterns of agents influence excitement and social presence in passive unidirectional communication scenarios.

Augmented reality (AR) presents vivid opportunities for interpersonal communication. With the growing diversity of social AR users, understanding their unique needs and perceptions becomes crucial. This study delves into how younger, middle-aged, and older adults perceive avatars with different aging attributes and degrees of realism, focusing on their anticipated user experience within a social AR system. We conducted an online within-subjects experiment involving $N = 2086$ age-diverse participants from Germany who assessed a set of nine gender-matched avatars for their perceived social attractiveness (research question 1 = RQ1) and the likelihood of selecting these avatars for self-representation in social AR (RQ2). The evaluated avatars represented different age groups (younger, middle-aged, and older) and levels of realism (low, medium, and high). We validated both the created avatars and our experimental setup and employed a linear mixed-effects modeling approach to analyze the data. Our findings unveiled a strong preference for younger high-realism avatars as communication partners (RQ1), which was consistent across all participant age groups. Similarly, participants favored younger high-realism avatars for self-representation in social AR (RQ2). However, older adults were more inclined to opt for avatars resembling their actual age. The study highlights the prevalence of age-related stereotypes in avatar-based communication. Similar to face-to-face social interactions, these stereotypes tend to render older avatars less socially attractive than their younger counterparts, irrespective of the avatar’s degree of realism. Our results invite considerations on how to combat these stereotypes through a more thoughtful and inclusive avatar design process that encompasses a broader spectrum of aging attributes.

Realistic lighting approaches typically rely on physically-based rendering which in turn often makes use of image-based lighting. Enabling these techniques in augmented reality on mobile devices requires unique approaches to estimate light probes, given the limited camera and sensor data available. In this paper, we evaluate different time-dependent and time-independent techniques for light probe estimation in augmented reality applications that try to predict the environment lighting using single images or video streams in combination with inpainting techniques. We simulate real-world applications using an evaluation framework where a simulated mobile device captures camera streams from different scenarios following a pre-defined path. The resulting camera streams are fed to a total of six estimation techniques in order to create light probes which are then used to render virtual objects while applying various materials. By comparing the rendered images as well as the light probe estimations, we perform a quantitative evaluation. We show how approaches that are able to process continuous video streams can provide more plausible results in cases where sufficient camera movement is present. Additionally, we investigate the visual impression of different types of materials showing that rough surfaces with distinct colors are less likely to produce divergent estimation results.

360-degree image data is a crucial aspect in graphics applications where they are typically used for lighting purposes. Fields like mixed reality generally rely on lighting estimation techniques to estimate the 360-degree environment. To evaluate such approaches, accurate image assessment in this domain is important. However, traditional image evaluation metrics like SSIM, PSNR, and IMED are problematic when analyzing 360-degree image data as they would require two-dimensional representations like equirectangular panoramas or unfolded cubes that introduce image distortions. In this paper, we address this problem by presenting Cube-SSIM, a variant of SSIM designed specifically for cube maps. For this, we modify SSIM to take the solid angles of cube map pixels into account which gives more consistent results than using SSIM for the individual cube faces. The computations can run efficiently on graphics hardware due to their native support for cube maps and no further image conversions are required. We show that our approach allows for more accurate results than other comparison metrics that largely depend on 2D images. While SSIM is especially important due to its wide usage, the modification can also be applied to other image metrics for which we include IMED as an example.

Previous studies have explored the perception of various types of embodied avatars in immersive environments. However, the impact of eye and face tracking with personalized avatars is yet to be explored. In this paper, we investigate the impact of eye and face tracking on embodiment, enfacement, and the uncanny valley with four types of avatars using a VR-based mirroring task. We conducted a study (N=12) and created self-avatars with two rendering styles: a cartoon avatar (created in an avatar generator using a picture of the user’s face) and a photorealistic scanned avatar (created using a 3D scanner), each with and without eye and face tracking and respective adaptation of the mirror image. Our results indicate that adding eye and face tracking can be beneficial for certain enfacement scales (belonged), and we confirm that compared to a cartoon avatar, a scanned realistic avatar results in higher body ownership and increased enfacement (own face, belonging, mirror) — regardless of eye and face tracking. We critically discuss our experiences and outline the limitations of the applied hardware and software with respect to the provided level of control and the applicability for complex tasks such as displaying emotions. We synthesize these findings into a discussion about potential improvements for facial animation in VR and highlight the need for a better level of control, the integration of additional sensing and processing technologies, and an objective metric for comparing facial animation systems.

The combination of smartphone Augmented Reality (AR) and Virtual Reality (VR) makes it possible for on-site and remote users to simultaneously explore a physical space and its digital twin through an asymmetric Collaborative Virtual Environment (CVE). In this paper, we investigate two spatial awareness visualizations to enable joint exploration of a space for dyads consisting of a smartphone AR user and a head-mounted display VR user. Our study revealed that both, a mini-map-based method and an egocentric compass method with a path visualization, enabled the on-site visitors to locate and follow a virtual companion reliably and quickly. Furthermore, the embodiment of the AR user by an inverse kinematics avatar allowed the use of natural gestures such as pointing and waving which was preferred over text messages by the participants of our study. In an expert review in a museum and its digital twin we observed an overall high social presence for on-site AR and remote VR visitors and found that the visualizations and the avatar embodiment successfully facilitated their communication and collaboration.

Augmented Reality (AR) and Virtual Reality (VR) are pushing from the labs towards consumers, especially with social applications. These applications require visual representations of humans and intelligent entities. However, displaying and animating photo-realistic models comes with a high technical cost while low-fidelity representations may evoke eeriness and overall could degrade an experience. Thus, it is important to carefully select what kind of avatar to display. This article investigates the effects of rendering style and visible body parts in AR and VR by adopting a systematic literature review. We analyzed 72 papers that compare various avatar representations. Our analysis includes an outline of the research published between 2015 and 2022 on the topic of avatars and agents in AR and VR displayed using head-mounted displays, covering aspects like visible body parts (e.g., hands only, hands and head, full-body) and rendering style (e.g., abstract, cartoon, realistic); an overview of collected objective and subjective measures (e.g., task performance, presence, user experience, body ownership); and a classification of tasks where avatars and agents were used into task domains (physical activity, hand interaction, communication, game-like scenarios, and education/training). We discuss and synthesize our results within the context of today's AR and VR ecosystem, provide guidelines for practitioners, and finally identify and present promising research opportunities to encourage future research of avatars and agents in AR/VR environments.

Integrating taste in AR/VR applications has various promising use cases - from social eating to the treatment of disorders. Despite many successful AR/VR applications that alter the taste of beverages and food, the relationship between olfaction, gustation, and vision during the process of multisensory integration (MSI) has not been fully explored yet. Thus, we present the results of a study in which participants were confronted with congruent and incongruent visual and olfactory stimuli while eating a tasteless food product in VR. We were interested (1) if participants integrate bi-modal congruent stimuli and (2) if vision guides MSI during congruent/incongruent conditions. Our results contain three main findings: First, and surprisingly, participants were not always able to detect congruent visual-olfactory stimuli when eating a portion of tasteless food. Second, when confronted with tri-modal incongruent cues, a majority of participants did not rely on any of the presented cues when forced to identify what they eat; this includes vision which has previously been shown to dominate MSI. Third, although research has shown that basic taste qualities like sweetness, saltiness, or sourness can be influenced by congruent cues, doing so with more complex flavors (e.g., zucchini or carrot) proved to be harder to achieve. We discuss our results in the context of multimodal integration, and within the domain of multisensory AR/VR. Our results are a necessary building block for future human-food interaction in XR that relies on smell, taste, and vision and are foundational for applied applications such as affective AR/VR.

Smartphone Augmented Reality (AR) has already provided us with a plethora of social applications such as Pokemon Go or Harry Potter Wizards Unite. However, to enable smartphone AR for social applications similar to VRChat or AltspaceVR, proper user tracking is necessary to accurately animate the avatars. In Virtual Reality (VR), avatar tracking is rather easy due to the availability of hand-tracking, controllers, and HMD whereas smartphone AR has only the back-(and front) camera and IMUs available for this task. In this paper we propose ARIKA, a tracking solution for avatars in smartphone AR. ARIKA uses tracking information from ARCore to track the users hand position and to calculate a pose using Inverse Kinematics (IK). We compare the accuracy of our system against a commercial motion tracking system and compare both systems with respect to sense of agency, self-location, and body-ownership. For this, 20 participants observed their avatars in an augmented virtual mirror and executed a navigation and a pointing task. Our results show that participants felt a higher sense of agency and self location when using the full body tracked avatar as opposed to IK avatars. Interestingly and in favor of ARIKA, there were no significant differences in body-ownership between our solution and the full-body tracked avatars. Thus, ARIKA and it’s single-camera approach is valid solution for smartphone AR applications where body-ownership is essential.

Abstract When operating a conditionally automated vehicle, humans occasionally have to take over control. If the driver is out of the loop, a certain amount of time is necessary to gain situation awareness. This work evaluates the potential of stereoscopic 3D (S3D) dashboards for presenting smart S3D take-over-requests (TORs) to support situation assessment. In a driving simulator study with a 4 × 2 between-within design, we presented 3 smart TORs showing the current traffic situation and a baseline TOR in 2D and S3D to 52 participants doing the n-back task. We further investigate if non-standard locations affect the results. Take-over performance indicates that participants looked at and processed the TORs’ visual information and by that, could perform more safe take-overs. S3D warnings in general, as well as warnings appearing at the participants’ focus of attention and warnings at the instrument cluster, performed best. We conclude that visual warnings, presented on an S3D dashboard, can be a valid option to support take-over while not increasing workload. We further discuss participants’ gaze behavior in the context of visual warnings for automotive user interfaces.

Abstract In recent years, various depth sensors that are small enough to be used with mobile hardware have been introduced. They provide important information for use cases like 3D reconstruction or in the context of augmented reality where tracking and camera data alone would be insufficient. However, depth sensors may not always be available due to hardware limitations or when simulating augmented reality applications for prototyping purposes. In these cases, different approaches like stereo matching or depth estimation using neural networks may provide a viable alternative. In this paper, we therefore explore the imitation of depth sensors using deep neural networks. For this, we use a state-of-the-art network for depth estimation and adapt it in order to mimic a Structure Sensor as well as an iPad LiDAR sensor. We evaluate the network which was pre-trained on NYU V2 directly as well as several variations where transfer learning is applied in order to adapt the network to different depth sensors while using various data preprocessing and augmentation techniques. We show that a transfer learning approach together with appropriate data processing can enable an accurate modeling of the respective depth sensors.

Depth estimation is one of the basic building blocks for scene understanding. In the case of monocular depth estimation using neural networks, many such approaches are highly hardware dependent because they result in a task- and environment-specific optimizing problem. Most DNN methods use commonly available datasets which leads to overfitting on particular sensor properties. Finding a generalized model with the consideration of different hardware properties of sensors and platforms is challenging if not impossible. For this reason, it is desirable to adapt existing and well-trained models into a new domain in order to let them simulate different depth sensors without the need for large datasets and time-consuming learning. Therefore, a small dataset has been created with the Structure Sensor for evaluating the transferable structural characteristic between neural networks. Finally, two input feature representations for the neural networks are considered to mimic the depth sensor including its artifacts including holes. The results show that a simple domain adaptation technique and a small dataset are adequate to simulate and adapt to a specific domain from a target domain. Therefore, the network is able to accurately predict depth maps as if they were created by a specific depth sensor. This also includes unique artifacts of the sensor, thereby allowing for a plausible simulation of specific depth sensing hardware which is beneficial for areas like prototyping in the context of Augmented Reality.

Semantic segmentation aims at providing a fine-grained image prediction by assigning each pixel to a specific semantic category. Convolutional neural networks offer significant benefits for solving this problem. However, the success of such networks is closely related to the availability of corresponding data sets. To facilitate semantic segmentation in a broader range of scenarios, such as augmented reality in outdoor environments or universal image-to-image translation, adequate training data sets are necessary. We present OUTSIDE15k, a large-scale data set for semantic segmentation of universal outdoor scenes. The data is labeled with 24 different semantic classes. The images contain multiple outdoor scenarios and cover a variety of different resolutions. Additionally, we present OUTSIDE-Net, an improved neural network architecture integrating multi-level pooling, feature fusion, and a spatial mask for semantic segmentation of universal outdoor scenes. It extracts spatial and semantic features from the input images to perform the segmentation. With the presented data set, we show the capability of our network which outperforms state-of-the-art approaches by achieving up to 91.5% pixel accuracy.

Recent developments in augmented reality for TV productions encouraged broadcasters to enhance interaction with virtual content for moderators. However, traditional interaction methods are considered distracting and not intuitive. To overcome these issues, we performed a gesture elicitation study with a follow-up evaluation. For this, we considered TV moderators as primary users of the gestures as well as viewers as recipients. The elicited gesture set consists of five gestures for two types of camera shots (long shot and close shot). Findings of the evaluation study indicate that the derived set of gestures requires low physical and concentration effort from moderators. Also, both moderators and viewers found them appropriate to be used in TV with respect to understandability, distraction, likeability, and appropriateness. Using these gestures would allow moderators to control AR content in TV and tell stories in a modern and more expressive way.

Many virtual reality (VR) applications rely on passive haptics where virtual objects have a real counterpart that provides tactile feedback. In addition to that, many VR applications do not provide accurate hand representations, especially when there is a high chance of occlusion as this makes vision-based tracking problematic. Hence, we investigated how a simple hand representation affects user experience and presence when interacting with passive haptic controls in a virtual environment. We report on a between-subject user study where N = 45 participants experienced one of three conditions (no hands at all, hands represented as a rigid 3D model, and hands represented as a rigid 3D model with a snapping mechanism). Our results indicate that a simple hand representation using a 3D model of hands paired with a snapping mechanism significantly increases presence and user experience. That indicates that this simple and low-cost technique is effective to improve the VE as a whole. This, in return, provides a chance for improvement for many VR applications with passive haptics.

State-of-the-art methods for diminished reality propagate pixel information from a keyframe to subsequent frames for real-time inpainting. However, these approaches produce artifacts, if the scene geometry is not sufficiently planar. In this article, we present InpaintFusion, a new real-time method that extends inpainting to non-planar scenes by considering both color and depth information in the inpainting process. We use an RGB-D sensor for simultaneous localization and mapping, in order to both track the camera and obtain a surfel map in addition to RGB images. We use the RGB-D information in a cost function for both the color and the geometric appearance to derive a global optimization for simultaneous inpainting of color and depth. The inpainted depth is merged in a global map by depth fusion. For the final rendering, we project the map model into image space, where we can use it for effects such as relighting and stereo rendering of otherwise hidden structures. We demonstrate the capabilities of our method by comparing it to inpainting results with methods using planar geometric proxies.

Augmented reality applications become more and more important to enhance our daily life and workflows. The possibility of showing additional virtual content in a camera stream is helpful for many different use cases like room planning where such applications can offer a simple and intuitive visualization. However, augmented reality applications can suffer from the interference by real objects that may disrupt the user experience. In recent years, there has been research regarding the removal of real objects from camera streams by applying diminished reality techniques. Current approaches are generally limited to flat objects, video streams with little camera movement, or can only remove objects in front of simple and mostly planar backgrounds. In our approach, we show a robust and efficient way to remove a selected 3D object from the camera stream visually. The removal is based on a dense 3D reconstruction of the physical environment stored in a voxel grid that can be created and extended on-the-fly. Hereby, an undesired object can be replaced by a background rendered from the reconstruction allowing for more complex environments than previous approaches. Furthermore, remaining holes by the removal of the object are removed applying an inpainting approach. Finally, we apply color correction to get a seamless transition between the virtual content and the camera image.

In recent years, stereoscopic 3D (S3D) displays have shown promising results on user experience, for navigation, and critical warnings when applied in cars. However, previous studies have only investigated these displays in non-interactive use-cases. So far, interacting with stereoscopic 3D content in cars has not been studied. Hence, we investigated how people interact with large S3D dashboards in automated vehicles (SAE level 4). In a user-elicitation study (N=23), we asked participants to propose interaction techniques for 24 referents while sitting in a driving simulator. Based on video recordings and motion tracking data of 1104 proposed interactions containing gestures and other input modalities, we grouped the gestures per task. Overall, we can report a chance-corrected agreement rate of k = 0.232 and by that, a medium agreement among participants. Based on the agreement rates, we defined two sets of gestures: a basic and a holistic version. Our results show that participants intuitively interact with S3D dashboards and that they prefer mid-air gestures that either directly manipulate the virtual object or operate on a proxy object. We further compare our results with similar results in different settings and provide insights on factors that have shaped our gesture set.

Glossy reflections of the surroundings play a major role when trying to achieve a seamless fusion of real and virtual objects in Mixed Reality (MR) environments. Traditionally, the necessary information about the ambiance is captured using mirrored balls, HDR cameras, fish-eye lenses, RGB-D cameras or 360-degree cameras. While these approaches allow for pretty good results, they require a rather complex setup. Our approach is based on a single RGB camera capturing the environmental lighting at a certain location within the scene. Therefore, we apply a precomputation step generating a 360-degree environment map and combine it with a camera-based image stitching for a continuous enhancement and update of the lighting information. We show that our approach allows for realistic and high-quality reflections within an AR/MR environment in real time even on mobile devices.

Real-time capable simultaneous localization and mapping (SLAM) approaches applying consumer hardware have been extensively researched in recent years. Their 3D reconstruction typically applies voxel volumes stored in regular grid hierarchies, sparse voxel octrees or voxel hash tables. They represent the model implicitly in the form of a truncated signed distance function (TSDF). Data integration is usually achieved by stepping through the reconstruction hierarchy from top to bottom and checking voxel grids against the new input data or by rasterizing input data to find associated voxels. For hierarchical representations, a major challenge remains the efficient determination of relevant portions of the reconstruction to be modified by new input data. We present a novel approach efficiently rasterizing input point clouds into intermediate volumes by the GPU. Our technique performs a simple preprocessing step on the input data to properly account for the TSDF representation, allowing for an accurate and hole-free reconstruction. We show that our approach is well suited for a fast integration of new input data into the hierarchical 3D reconstruction, allowing for real-time performance while only slightly increasing memory consumption.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

In recent years, virtual reality products have found their way into our life. They have changed our entertainment experience in many ways, including the way people consume adult entertainment. With the immersion and presence offered by VR technologies, a new level of user experience in adult entertainment might be possible. To explore these possibilities, we conducted a qualitative user study with 12 participants to investigate participants views on realism and interactivity in immersive adult content videos and a virtual reality erotic digital games. Results indicate that a high sense interactivity is no less important than a high level of visual realism.

Up to now, most driving simulators use either small monitors or large immersive projection setups like 2D/3D screens or a CAVE. The recent improvements of VR-HMDs led to an increased application in driving simulation. However, the influence and comparability of various VR and non-VR displays has been hardly investigated. We present results of a user study investigating the different influence of non-VR (2D, stereoscopic 3D) and VR (HMD) on physiological responses, simulation sickness, and driving performance within a single driving simulator. In the study, 94 participants performed the Lane Change Task. Results indicate that a VR-HMD leads to similar data as stereoscopic 3D or 2D screens. We observed no significant difference regarding physiological responses or lane change performance. However, we measured significantly increased simulator sickness in the VR-HMD condition compared to stereoscopic 3D.

Proper scene inference provides the basis for a seamless integration of virtual objects into the real environment. While widely neglected in many AR/MR environments, previous approaches providing good results were based on rather complex setups, often involving mirrored balls, several HDR cameras, and fish eye lenses to achieve proper light probes. In this paper we present an approach requiring a single RGB-D camera image only for generating glossy reflections on virtual objects. Our approach is based on a partial 3D reconstruction of the real environment combined with a screen-space ray-tracing mechanism. We show that our approach allows for convincing reflections of the real environment as well as mutual reflections between virtual objects of an MR environment.

While image inpainting has recently become widely available in image manipulation tools, existing approaches to video inpainting typically do not even achieve interactive frame rates yet as they are highly computationally expensive. Further, they either apply severe restrictions on the movement of the camera or do not provide a high-quality coherent video stream. In this paper we will present our approach to high-quality real-time capable image and video inpainting. Our PixMix approach even allows for the manipulation of live video streams, providing the basis for real Diminished Reality (DR) applications. We will show how our approach generates coherent video streams dealing with quite heterogeneous background environments and non-trivial camera movements, even applying constraints in real-time.

While global illumination is a crucial issue for most computer graphics applications fostering photo realistic rendering, fast and efficient implementations remain challenging for real-time applications. One approach to approximate indirect illumination is to distribute virtual point lights (VPL) at surfaces that emit indirect light. This distribution may be realized using reflective shadow maps (RSM). One major drawback of this approach is that each surface point has to be illuminated by thousands of VPLs, leading to a performance bottleneck in the shading step. Therefore several approaches trying to reduce the shading costs either by decreasing the number of VPLs or by lowering the surface points to be shaded exist.

Designing Augmented Reality location aware games requires an understanding of how form and content issues impact on presence. A study of 60 players was conducted using questionnaires, video analysis and interviews. The results indicate that content including: moral dilemmas, strong narratives, using real locations effectively and applying simple physical behavior within virtual characters to improve embodiment have a positive impact on player experience. The results are presented in the form of guidelines.

Diminished Reality (DR) allows to remove objects from a video stream while preseving a frame to frame coherence. Some approaches apply a pseudo-DR, allowing for the removal of objects only, while their background can be observed by a second camera. Most real DR approaches are highly computational expensive, not even allowing for interactive rates and/or apply significant restrictions regarding the uniformity of the background, or allow linear camera movements or even a static camera only. In this paper we will present a real-time capable Diminished Reality approach for high-quality image manipulation. Our approach achieves a significantly better performance and image quality for almost planar but non-trivial image backgrounds. Our Diminished Reality pipeline provides coherent video streams even for nonlinear camera movements due to the integration of homography based object tracking.

For seamless integration of virtual content into real scenes, realizing mutual global lighting effects between both worlds belongs to the most important and challenging goals. Therefore, plenty of global illumination approaches exist, which mostly share the same restriction: the real scene is approximated by a static model, which was built in advance and thus has to remain static. In our paper, we propose an image-space global illumination approach, based on reflective shadow maps, combined with the use of an RGB-D camera, to simulate first bounce diffuse indirect illumination without any pre-computations. Our approach supports indirect illumination in both directions (real to virtual and vice versa) and runs in real-time. Furthermore, it does not require advanced shader properties, since we developed an implementation making efficient usage of the Z-Buffer algorithm for calculating indirect illumination.

Feature based tracking approaches become more and more common for Augmented Reality (AR). However, most upcoming AR solutions are designed for mobile devices, in particular for smartphones and tablet computers, lacking sufficient performance for the execution of state-of-the art feature based approaches at interactive frame rates. In this paper we will present our approach significantly increasing the speed of feature based tracking, thus allowing for real-time applications even on mobile devices. Our approach applies a randomized pose initialization, is applicable to any feature detector and does not require any feature appearance attributes, such as descriptors or ferns.

The seamless integration of the real and the virtual content is the ultimate yet unreached goal of Mixed Reality applications. Among others it requires mutual blocking and lighting between real and virtual objects. In this paper we present our approach of applying a low-cost depth camera, such as Kinect, allowing for an easy acquisition of depth images. However, as the quality of the raw input data is insufficient for this purpose, we apply a series of filter and optimization operations. This allows us to realize mutual real-time lighting and rigid interaction in a dynamic environment. Our approach produces an acceptable quality of images of low-frequency scenes at interactive frame rates on an off-the-shelf desktop computer.

We present a novel technique implementing barehanded interaction with virtual 3D content by employing a time-of-flight camera. The system improves on existing 3D multi-touch systems by working regardless of lighting conditions and supplying a working volume large enough for multiple users. Previous systems were limited either by environmental requirements, working volume, or computational resources necessary for realtime operation. By employing a time-of-flight camera, the system is capable of reliably recognizing gestures at the finger level in real-time at more than 50 fps with commodity computer hardware using our newly developed precision hand and finger-tracking algorithm. Building on this algorithm, the system performs gesture recognition with simple constraint modeling over statistical aggregations of the hand appearances in a working volume of more than 8 cubic meters. Two iterations of user tests were performed on a prototype system, demonstrating the feasibility and usability of the approach as well as providing first insights regarding the acceptance of true barehanded touch-based 3D interaction.

While Augmented Reality has always been restricted to adding artificial content to the real environment, Diminished Reality allows for removing real world content. Existing approaches however, either require complex setups or are not applicable in real-time. In this paper we present our approach for removing real-world objects from a live video stream of the user's real environment. Our approach is based on a simple setup and neither requires any pre-processing nor any information on the structure and location of the objects to be removed or on their background. Our approach is based on the identification of the objects to be removed combined with an image completion and synthesis algorithm. The performance of our approach is one to two magnitudes better than that of previous work in the area of image completion, providing real-time object cancellation on standard laptop or tablet computers.

While tracking technologies based on fiducial markers have dominated the development of Augmented Reality (AR) applications for almost a decade, various real-time capable approaches to markerless tracking have recently been presented. However, most existing approaches do not yet achieve sufficient frame rates for AR on mobile phones or at least require an extensive training phase in advance.

Mobile location-aware applications have become quite popular across a range of new areas such as pervasive games and mobile edutainment applications. However it is only recently, that approaches have been presented which combine gaming and education with mobile Augmented Reality systems. However they typically lack a close crossmedia integration of the surroundings, and often annotate or extend the environment rather than modifying and altering it.

In this paper we present a pervasive outdoor mixed reality edutainment game for exploring the history of a city in the spatial and the temporal dimension, which will closely couple the real environment with the virtual content. The game provides a new and unique user experience, which links rich interactive content to time and places. We introduce the development of such a game, including a universal mechanism to define and setup multi-modal user interfaces for game challenges.

High-end mobile devices are becoming increasing popular in every day life. Augmented reality (AR) builds on this trend by combining mobile computing with connectivity and location-awareness. In doing so, AR can provide a very rich user experience. In this paper we discuss the approach and development of an AR-based personal assistant, combining the familiar interface of a human person with the functionality of a location-aware digital information system. The paper discusses the main components of the system, including the anthropomorphic user interface as well as the results of an initial prototype evaluation.

Render engines or render APIs are a core part of each virtual reality (VR) or augmented reality (AR) environment as well as 3D games. Most existing approaches either focus on rendering speed for high frame rates or on the presentation of advanced visual features. However, no approach exists to integrate scene descriptions based on multiple file formats without converting and thereby partly destroying the native scene format. In this paper we will present our approach of internal and external scene graphs for realizing render engines. We will show how this approach overcomes existing limitations, while still providing decent frame rates and rendering features. By using internal scene graphs for rendering, we use external scene graphs to store format or application specific information, preserving its native structure and content.

Augmented Reality (AR) technologies are rapidly expanding into new application areas. However, the development of AR user interfaces and appropriate interaction techniques remains a complex and time-consuming task. Starting from scratch is more common than building upon existing solutions. Furthermore, adaptation is difficult, often resulting in poor quality and limited flexibility with regard to user requirements. In order to overcome these problems, we introduce an infrastructure for supporting the development of specific AR interaction techniques and their adaptation to individual user needs. Our approach is threefold: a flexible AR framework providing independence from particular input devices and rendering platforms, an interaction prototyping mechanism allowing for fast prototyping of new interaction techniques, and a high-level user interface description, extending user interface descriptions into the domain of AR. The general usability and applicability of the approach is demonstrated by means of three example AR projects.

The availability of a suitable framework is of vital importance for the development of Augmented Reality (AR) and Virtual Reality (VR) projects. While features such as scalability, platform independence, support of multiple users, distribution of components, and an efficient and sophisticated rendering are the key requirements of current and future applications, existing frameworks often address these issues only partially. In our paper we present MORGAN --- an extensible component-based AR/VR framework, enabling sophisticated dynamic multi-user AR and VR projects. Core components include the MORGAN API, providing developers access to various input devices, including common tracking devices, as well as a modular render engine concept, allowing us to provide native support for individual scene graph concepts. The MORGAN framework has already been successfully deployed in several national and international research and development projects.