Current and, above all, future mixed reality services and systems collect a large amount of different personal and environmental data of the users and other involved persons. Current legal regulations on data protection and governance do not appear to adequately meet these challenges. This paper looks at data collection in Mixed Reality systems in order to create a greater awareness for its far-reaching privacy-related implications. We present a straightforward scenario where a user named Melanie interacts with both her environment and another user. It is stipulated that Melanie is not a first-time user, and all interactions and environmental factors are predetermined by an existing data profile. This paper examines the technical design of the scenario, the types of data collected, and their legal implications. To address the far-reaching privacy implications and shortcomings in the current legal framework, we propose technical solutions for less privacy-encroaching data uses and recommend the implementation of better legal instruments and safeguards in the future. In conclusion we suggest that better legal instruments and safeguards are needed in future.

Augmented Reality (AR) and Virtual Reality (VR) are essential tools for researchers and practitioners, serving purposes from training to entertainment: many of these applications rely on agents. This study explores the impact of agent characteristics on user reactions, focusing on gaze as a primary visual attention indicator in AR and VR. While existing research has investigated the agent’s gaze and its influence on the user, it is unclear how the agent’s auralization and visualization influence gaze behaviour. We investigate this by studying the impact of rendering style and type of audio on gaze behaviour during a narrative AR experience. Participants listened to a story with the agent visualized as a cartoon-style or realistic virtual human and auralized with spatial or non-spatial audio. The results revealed that the agent’s rendering style significantly influenced gaze behaviour, with cartoon-style agents capturing more visual attention. Audio variations did not yield significant differences. Together, our findings inform the design of AR user interfaces with agents, suggesting that low-realism visualizations are more captivating and, thus, more suitable for experiences where the user is supposed to look at the storyteller.

In unusual environments and situations like extreme sports, underground environments, underwater, conflict zones or outer space, human sensory perception is often limited, which can adversely affect task performance, situational awareness, and the feeling of presence. This paper explores the efficacy of augmenting human perception with audiovisual information to support individuals in such contexts. Specifically, we investigate the impact of audiovisual augmentation on presence, situational awareness, and task performance. To conduct this research, we designed a virtual reality (VR) simulation of a space mission aboard the International Space Station (ISS). Within this simulation, participants were tasked with performing maintenance activities while receiving artificial augmentations. We conducted a user study involving 43 participants who performed the same maintenance task under four augmentation conditions: audio cues, visual cues, audiovisual cues, and no cues. Our findings reveal that adding audiovisual information significantly enhances performance. Participants with audiovisual augmentations had a 60% improvement in task completion speed compared to those without augmentations. Moreover, the workload was substantially reduced, and the sense of presence was increased when audiovisual support was used. The results highlight the potential of audiovisual augmentation as a valuable tool for individuals engaging in situations with reduced audiovisual perception. The insights demonstrating the versatility and promise of audiovisual augmentation in enhancing task performance.

The appearance of virtual humans (avatars and agents) has been widely explored in immersive environments. However, virtual humans’ movements and associated sounds in real-world interactions, particularly in Augmented Reality (AR), are yet to be explored. In this paper, we investigate the influence of three distinct movement patterns (circle, side-to-side, and standing), two rendering styles (realistic and cartoon), and two types of audio (spatial audio and non-spatial audio) on emotional responses, social presence, appearance and behavior plausibility, audiovisual coherence, and auditory plausibility. To enable that, we conducted a study (N=36) where participants observed an agent reciting a short fictional story. Our results indicate an effect of the rendering style and the type of movement on the subjective perception of the agents behaving in an AR environment. Participants reported higher levels of excitement when they observed the realistic agent moving in a circle compared to the cartoon agent or the other two movement patterns. Moreover, we found an influence of agent’s movement pattern on social presence and higher appearance and behavior plausibility for the realistic rendering style. Regarding audiovisual spatial coherence, we found an influence of rendering style and type of audio only for the cartoon agent. Additionally, the spatial audio was perceived as more plausible than non-spatial audio. Our findings suggest that aligning realistic rendering styles with realistic auditory experiences may not be necessary for 1-1 listening experiences with moving sources. However, movement patterns of agents influence excitement and social presence in passive unidirectional communication scenarios.

This contribution presents a summary of results from perceptual auditory experiments on context dependent quality parameters for virtual acoustic environments. The investigated quality features are influenced by divergence between synthesized scene and listening room and adaptation on congruence or divergence between the rooms. Two experiments are presented. The results from the first experiment show the room divergence effect on spatial auditory perception. A divergence between the listening room and binaurally synthesized room leads to a decrease of perceived externalization while congruence yields an increase. A more comprehensive statistical analysis regarding significance, effect size and visual influences is added to complement the original publication of this data. The second experiment shows this effect as the result of expectations of the listeners and can be shifted by adaptation and training. In the experiments we show, that training to congruent or divergent room combinations can increase or decrease the room divergence effect.