Older adults, a growing demographic, face an increased risk of experiencing loneliness and are less exposed to emerging communication technologies. Augmented reality (AR) avatars and telepresence robots have been proposed as tools to foster social connection, yet their suitability for older users remains underexplored. We present an exploratory study with ten healthy older adults who engaged in both conversational and spatial collaboration tasks using AR avatar-mediated communication, robot-mediated communication, and face-to-face interaction. We collected self-reported measures of co-presence, social presence, closeness, uncanny valley, preferences, and open feedback. Our findings suggest that telepresence robots enhanced co-presence, while avatars were valued for their expressivity and humanlike qualities. Task type influenced co-presence in spatial collaboration only during communication using the telepresence robot. Other measures, such as social presence and closeness, were unaffected by task type or representation. While neither technology outperformed face-to-face interaction, both were positively received, underscoring their potential to address the social needs of older adults and highlighting the importance of enhancing nonverbal expressivity, particularly nonverbal cues in mediated communication. Ultimately, our results contribute to the fundamental understanding of mediated communication with older adults, motivating further empirical work to confirm and extend these findings.

Video pass-through (VPT) techniques often cause visual distortions like the ‘telescope eye' effect, which leads to less immersion and spatial awareness in Extended Reality (XR). This study investigates the potential of 3D Gaussian Splatting (3DGS) as a rendering alternative to video pass-through, particularly focusing on its ability to mitigate common perceptual artifacts and enhance immersion. To evaluate the efficacy of 3DGS in an XR context, we conducted a user study comparing two visualization types: (1) conventional VPT and (2) 3DGS-based scene representation. Three distinct scenes with high detail, natural depth perception, and reflective surfaces were included in the study. An empirical study was conducted with a structured questionnaire that inquired about the level of immersion and the impact of the visual stimuli. Our results provide insights into the scalability and hardware considerations for implementing 3DGS in realtime interactive environments. The evaluation results indicated that users with a 3DGS visualization experienced a reduction in visual distortion and an enhancement in the perceived realism of objects. The findings show that 3DGS can serve as an effective alternative to VPT, potentially improving realism and reducing perceptual artifacts in XR applications.

Avatars and telepresence robots have long received attention for remote communication. However, the specific nature of their physicality, expressiveness, and mobility may affect their usefulness for different tasks. This work compares using an avatar (presented in augmented reality) and a telepresence robot to Face-to-Face (F2F) communication during different communication tasks: free conversation, negotiation, and referential communication with movement. We conducted a user study (split-plot design, N=54) with the type of representation of the conversational partner as the within variable and the communication task as the between variable. Our results show that the type of task, especially referential communication with movement, influenced the perceived attention to nonverbal cues and closeness. Generally, gestures and body movements received the least focus with telepresence robots. Gestures in avatars and F2F drew similar attention, which we attribute to the avatar's tracking fidelity. Gaze received less attention in both avatar- and robot-mediated communication compared to F2F, while facial expressions on the robot's screen heightened attention compared to avatars. These findings advance the fundamental understanding of mediated communication and support researchers and practitioners in shaping the design of communication applications beyond today's video calls.

Abstract To achieve realistic lighting in augmented reality (AR), virtual reality (VR), and similar applications especially with stringent realtime constraints, environmental lighting is often captured and stored in light probes, which are strategically placed throughout the scene. This study explores the optimal positioning of light probes to enhance the realism of extended realities. A novel spatial geometry-based method for light probe placement is proposed and compared against three alternative techniques. They are evaluated across various 3D Gaussian splatting environments with different probe distributions. The techniques are compared to ground truth renderings that are obtained using a pipeline that combines fast Gaussian rasterization with ray tracing. The findings demonstrate that the careful placement of a minimal number of probes can effectively replicate realistic lighting in complex environments. The geometry-based optimization method outperforms the alternatives in indoor environments by providing an efficient realtime solution with fewer probes. This optimized approach is suitable for direct implementation in realtime applications such as AR, VR, game development, and virtual productions, offering both improved lighting realism and optimized computational efficiency.

In order to achieve realistic lighting in augmented reality (AR), virtual reality (VR), and other virtual environments with stringent real-time requirements, environmental lighting is captured and stored in light probes that are strategically positioned throughout these environments. This study investigates the optimal positioning of light probes based on the environment to achieve realistic virtual representations. A novel spatial geometry-based approach for light probe placement is introduced, validated in a comparative analyses against three alternative methods across diverse virtual environments, each utilizing different probe quantities. The investigation reveals that strategic deployment of a minimal number of probes can effectively simulate realistic illumination across complex environmental scenarios. The geometry-based optimization outperforms alternative methods by achieving efficient real-time probe placement with minimal number of probes. This optimized methodology can be directly implemented at runtime in various applications, including AR, VR, game development, and virtual productions, thereby enhancing lighting realism while optimizing computational resources.

Augmented Reality (AR) and Virtual Reality (VR) are essential tools for researchers and practitioners, serving purposes from training to entertainment: many of these applications rely on agents. This study explores the impact of agent characteristics on user reactions, focusing on gaze as a primary visual attention indicator in AR and VR. While existing research has investigated the agent’s gaze and its influence on the user, it is unclear how the agent’s auralization and visualization influence gaze behaviour. We investigate this by studying the impact of rendering style and type of audio on gaze behaviour during a narrative AR experience. Participants listened to a story with the agent visualized as a cartoon-style or realistic virtual human and auralized with spatial or non-spatial audio. The results revealed that the agent’s rendering style significantly influenced gaze behaviour, with cartoon-style agents capturing more visual attention. Audio variations did not yield significant differences. Together, our findings inform the design of AR user interfaces with agents, suggesting that low-realism visualizations are more captivating and, thus, more suitable for experiences where the user is supposed to look at the storyteller.

Given the profound impact of staying socially connected on the well-being of older adults, this study explores the potential of augmented reality (AR) systems to enrich their social lives. A wearable AR communication system prototype was developed and tested in a user study involving N = 16 older adults from Germany. Participants wore an AR headset and engaged in a conversation task with a remote person represented by an avatar. Older adults’ experiences were assessed using think-aloud protocols, qualitative observations, posttest questionnaires, and semi-structured oral interviews. Preliminary findings indicate overall participant satisfaction, with minimal observed difficulties in headset usage and avatar-mediated interpersonal communication. The positive engagement during AR conversations highlights the system’s potential to provide positive communication experiences among older individuals. This work-in-progress paper introduces the developed system prototype and outlines the conducted user study. Further data analyses will provide deeper insights into older adults’ experiences with the system. The results will contribute to refining the prototype and offer valuable insights for the development of AR communication systems tailored to the needs and preferences of older adults.

The appearance of virtual humans (avatars and agents) has been widely explored in immersive environments. However, virtual humans’ movements and associated sounds in real-world interactions, particularly in Augmented Reality (AR), are yet to be explored. In this paper, we investigate the influence of three distinct movement patterns (circle, side-to-side, and standing), two rendering styles (realistic and cartoon), and two types of audio (spatial audio and non-spatial audio) on emotional responses, social presence, appearance and behavior plausibility, audiovisual coherence, and auditory plausibility. To enable that, we conducted a study (N=36) where participants observed an agent reciting a short fictional story. Our results indicate an effect of the rendering style and the type of movement on the subjective perception of the agents behaving in an AR environment. Participants reported higher levels of excitement when they observed the realistic agent moving in a circle compared to the cartoon agent or the other two movement patterns. Moreover, we found an influence of agent’s movement pattern on social presence and higher appearance and behavior plausibility for the realistic rendering style. Regarding audiovisual spatial coherence, we found an influence of rendering style and type of audio only for the cartoon agent. Additionally, the spatial audio was perceived as more plausible than non-spatial audio. Our findings suggest that aligning realistic rendering styles with realistic auditory experiences may not be necessary for 1-1 listening experiences with moving sources. However, movement patterns of agents influence excitement and social presence in passive unidirectional communication scenarios.

Augmented reality (AR) presents vivid opportunities for interpersonal communication. With the growing diversity of social AR users, understanding their unique needs and perceptions becomes crucial. This study delves into how younger, middle-aged, and older adults perceive avatars with different aging attributes and degrees of realism, focusing on their anticipated user experience within a social AR system. We conducted an online within-subjects experiment involving $N = 2086$ age-diverse participants from Germany who assessed a set of nine gender-matched avatars for their perceived social attractiveness (research question 1 = RQ1) and the likelihood of selecting these avatars for self-representation in social AR (RQ2). The evaluated avatars represented different age groups (younger, middle-aged, and older) and levels of realism (low, medium, and high). We validated both the created avatars and our experimental setup and employed a linear mixed-effects modeling approach to analyze the data. Our findings unveiled a strong preference for younger high-realism avatars as communication partners (RQ1), which was consistent across all participant age groups. Similarly, participants favored younger high-realism avatars for self-representation in social AR (RQ2). However, older adults were more inclined to opt for avatars resembling their actual age. The study highlights the prevalence of age-related stereotypes in avatar-based communication. Similar to face-to-face social interactions, these stereotypes tend to render older avatars less socially attractive than their younger counterparts, irrespective of the avatar’s degree of realism. Our results invite considerations on how to combat these stereotypes through a more thoughtful and inclusive avatar design process that encompasses a broader spectrum of aging attributes.

Realistic lighting approaches typically rely on physically-based rendering which in turn often makes use of image-based lighting. Enabling these techniques in augmented reality on mobile devices requires unique approaches to estimate light probes, given the limited camera and sensor data available. In this paper, we evaluate different time-dependent and time-independent techniques for light probe estimation in augmented reality applications that try to predict the environment lighting using single images or video streams in combination with inpainting techniques. We simulate real-world applications using an evaluation framework where a simulated mobile device captures camera streams from different scenarios following a pre-defined path. The resulting camera streams are fed to a total of six estimation techniques in order to create light probes which are then used to render virtual objects while applying various materials. By comparing the rendered images as well as the light probe estimations, we perform a quantitative evaluation. We show how approaches that are able to process continuous video streams can provide more plausible results in cases where sufficient camera movement is present. Additionally, we investigate the visual impression of different types of materials showing that rough surfaces with distinct colors are less likely to produce divergent estimation results.

360-degree image data is a crucial aspect in graphics applications where they are typically used for lighting purposes. Fields like mixed reality generally rely on lighting estimation techniques to estimate the 360-degree environment. To evaluate such approaches, accurate image assessment in this domain is important. However, traditional image evaluation metrics like SSIM, PSNR, and IMED are problematic when analyzing 360-degree image data as they would require two-dimensional representations like equirectangular panoramas or unfolded cubes that introduce image distortions. In this paper, we address this problem by presenting Cube-SSIM, a variant of SSIM designed specifically for cube maps. For this, we modify SSIM to take the solid angles of cube map pixels into account which gives more consistent results than using SSIM for the individual cube faces. The computations can run efficiently on graphics hardware due to their native support for cube maps and no further image conversions are required. We show that our approach allows for more accurate results than other comparison metrics that largely depend on 2D images. While SSIM is especially important due to its wide usage, the modification can also be applied to other image metrics for which we include IMED as an example.

Previous studies have explored the perception of various types of embodied avatars in immersive environments. However, the impact of eye and face tracking with personalized avatars is yet to be explored. In this paper, we investigate the impact of eye and face tracking on embodiment, enfacement, and the uncanny valley with four types of avatars using a VR-based mirroring task. We conducted a study (N=12) and created self-avatars with two rendering styles: a cartoon avatar (created in an avatar generator using a picture of the user’s face) and a photorealistic scanned avatar (created using a 3D scanner), each with and without eye and face tracking and respective adaptation of the mirror image. Our results indicate that adding eye and face tracking can be beneficial for certain enfacement scales (belonged), and we confirm that compared to a cartoon avatar, a scanned realistic avatar results in higher body ownership and increased enfacement (own face, belonging, mirror) — regardless of eye and face tracking. We critically discuss our experiences and outline the limitations of the applied hardware and software with respect to the provided level of control and the applicability for complex tasks such as displaying emotions. We synthesize these findings into a discussion about potential improvements for facial animation in VR and highlight the need for a better level of control, the integration of additional sensing and processing technologies, and an objective metric for comparing facial animation systems.

Augmented Reality (AR) and Virtual Reality (VR) are pushing from the labs towards consumers, especially with social applications. These applications require visual representations of humans and intelligent entities. However, displaying and animating photo-realistic models comes with a high technical cost while low-fidelity representations may evoke eeriness and overall could degrade an experience. Thus, it is important to carefully select what kind of avatar to display. This article investigates the effects of rendering style and visible body parts in AR and VR by adopting a systematic literature review. We analyzed 72 papers that compare various avatar representations. Our analysis includes an outline of the research published between 2015 and 2022 on the topic of avatars and agents in AR and VR displayed using head-mounted displays, covering aspects like visible body parts (e.g., hands only, hands and head, full-body) and rendering style (e.g., abstract, cartoon, realistic); an overview of collected objective and subjective measures (e.g., task performance, presence, user experience, body ownership); and a classification of tasks where avatars and agents were used into task domains (physical activity, hand interaction, communication, game-like scenarios, and education/training). We discuss and synthesize our results within the context of today's AR and VR ecosystem, provide guidelines for practitioners, and finally identify and present promising research opportunities to encourage future research of avatars and agents in AR/VR environments.

Abstract In recent years, various depth sensors that are small enough to be used with mobile hardware have been introduced. They provide important information for use cases like 3D reconstruction or in the context of augmented reality where tracking and camera data alone would be insufficient. However, depth sensors may not always be available due to hardware limitations or when simulating augmented reality applications for prototyping purposes. In these cases, different approaches like stereo matching or depth estimation using neural networks may provide a viable alternative. In this paper, we therefore explore the imitation of depth sensors using deep neural networks. For this, we use a state-of-the-art network for depth estimation and adapt it in order to mimic a Structure Sensor as well as an iPad LiDAR sensor. We evaluate the network which was pre-trained on NYU V2 directly as well as several variations where transfer learning is applied in order to adapt the network to different depth sensors while using various data preprocessing and augmentation techniques. We show that a transfer learning approach together with appropriate data processing can enable an accurate modeling of the respective depth sensors.

Depth estimation is one of the basic building blocks for scene understanding. In the case of monocular depth estimation using neural networks, many such approaches are highly hardware dependent because they result in a task- and environment-specific optimizing problem. Most DNN methods use commonly available datasets which leads to overfitting on particular sensor properties. Finding a generalized model with the consideration of different hardware properties of sensors and platforms is challenging if not impossible. For this reason, it is desirable to adapt existing and well-trained models into a new domain in order to let them simulate different depth sensors without the need for large datasets and time-consuming learning. Therefore, a small dataset has been created with the Structure Sensor for evaluating the transferable structural characteristic between neural networks. Finally, two input feature representations for the neural networks are considered to mimic the depth sensor including its artifacts including holes. The results show that a simple domain adaptation technique and a small dataset are adequate to simulate and adapt to a specific domain from a target domain. Therefore, the network is able to accurately predict depth maps as if they were created by a specific depth sensor. This also includes unique artifacts of the sensor, thereby allowing for a plausible simulation of specific depth sensing hardware which is beneficial for areas like prototyping in the context of Augmented Reality.

Augmented reality applications become more and more important to enhance our daily life and workflows. The possibility of showing additional virtual content in a camera stream is helpful for many different use cases like room planning where such applications can offer a simple and intuitive visualization. However, augmented reality applications can suffer from the interference by real objects that may disrupt the user experience. In recent years, there has been research regarding the removal of real objects from camera streams by applying diminished reality techniques. Current approaches are generally limited to flat objects, video streams with little camera movement, or can only remove objects in front of simple and mostly planar backgrounds. In our approach, we show a robust and efficient way to remove a selected 3D object from the camera stream visually. The removal is based on a dense 3D reconstruction of the physical environment stored in a voxel grid that can be created and extended on-the-fly. Hereby, an undesired object can be replaced by a background rendered from the reconstruction allowing for more complex environments than previous approaches. Furthermore, remaining holes by the removal of the object are removed applying an inpainting approach. Finally, we apply color correction to get a seamless transition between the virtual content and the camera image.

Glossy reflections of the surroundings play a major role when trying to achieve a seamless fusion of real and virtual objects in Mixed Reality (MR) environments. Traditionally, the necessary information about the ambiance is captured using mirrored balls, HDR cameras, fish-eye lenses, RGB-D cameras or 360-degree cameras. While these approaches allow for pretty good results, they require a rather complex setup. Our approach is based on a single RGB camera capturing the environmental lighting at a certain location within the scene. Therefore, we apply a precomputation step generating a 360-degree environment map and combine it with a camera-based image stitching for a continuous enhancement and update of the lighting information. We show that our approach allows for realistic and high-quality reflections within an AR/MR environment in real time even on mobile devices.

Real-time capable simultaneous localization and mapping (SLAM) approaches applying consumer hardware have been extensively researched in recent years. Their 3D reconstruction typically applies voxel volumes stored in regular grid hierarchies, sparse voxel octrees or voxel hash tables. They represent the model implicitly in the form of a truncated signed distance function (TSDF). Data integration is usually achieved by stepping through the reconstruction hierarchy from top to bottom and checking voxel grids against the new input data or by rasterizing input data to find associated voxels. For hierarchical representations, a major challenge remains the efficient determination of relevant portions of the reconstruction to be modified by new input data. We present a novel approach efficiently rasterizing input point clouds into intermediate volumes by the GPU. Our technique performs a simple preprocessing step on the input data to properly account for the TSDF representation, allowing for an accurate and hole-free reconstruction. We show that our approach is well suited for a fast integration of new input data into the hierarchical 3D reconstruction, allowing for real-time performance while only slightly increasing memory consumption.

The lymphatic system is responsible for transporting interstitial fluid back to the bloodstream, but unlike the cardiovascular system, lacks a centralized pump-the heart-to drive flow. Instead, each collecting lymphatic vessel can individually contract and dilate producing unidirectional flow enforced by intraluminal check valves. Due to the large number and spatial distribution of such pumps, high-level coordination would be unwieldy. This leads to the question of how each segment of lymphatic vessel responds to local signals that can contribute to the coordination of pumping on a network basis. Beginning with elementary fluid mechanics and known cellular behaviors, we show that two complementary oscillators emerge from i) mechanical stretch with calcium ion transport and ii) fluid shear stress induced nitric oxide production (NO). Using numerical simulation and linear stability analysis we show that the newly identified shear-NO oscillator shares similarities with the well-known Van der Pol oscillator, but has unique characteristics. Depending on the operating conditions, the shear-NO process may i) be inherently stable, ii) oscillate spontaneously in response to random disturbances or iii) synchronize with weak periodic stimuli. When the complementary shear-driven and stretch-driven oscillators interact, either may dominate, producing a rich family of behaviors similar to those observed in vivo.