We present Editing Reality, a mixed reality system that enables in-situ co-creation with generative AI directly within physical environments. Rather than treating generation as a one-shot command, the system supports embodied and iterative creation through speech, sketching, and direct manipulation, allowing users to generate, modify, erase, and retexture real-anchored virtual and reconstructed scene elements in place. Using a Research Through Design approach, we investigate how co-creation unfolds through iterative system development, a formative workshop, and expert review. From this process, we articulate a set of designerly framings that characterize in-situ co-creation as a negotiated, spatial, and temporal practice shaped by previews, accumulation, waiting, embodied evaluation, and learning the system as a spatial actor. We instantiate these ideas in a working system and report expert feedback highlighting both its creative potential and its design implications. Our work contributes a conceptual lens for understanding generative AI in mixed reality not as a one-shot automation tool, but as part of an embodied, situated creative process.

In-person human interaction relies on our spatial perception of each other and our surroundings. Current remote communication tools partially address each of these aspects. Video calls convey real user representations but without spatial interactions. Augmented and Virtual Reality (AR/VR) experiences are immersive and spatial but often use virtual environments and characters instead of real-life representations. Bridging these gaps, we introduce DualStream, a system for synchronous mobile AR remote communication that captures, streams, and displays spatial representations of users and their surroundings. DualStream supports transitions between user and environment representations with different levels of visuospatial fidelity, as well as the creation of persistent shared spaces using environment snapshots. We demonstrate how DualStream can enable spatial communication in real-world contexts, and support the creation of blended spaces for collaboration. A formative evaluation of DualStream revealed that users valued the ability to interact spatially and move between representations, and could see DualStream fitting into their own remote communication practices in the near future. Drawing from these findings, we discuss new opportunities for designing more widely accessible spatial communication tools, centered around the mobile phone.

Augmented Reality (AR), with its ability to make people feel like they are in the same space as friends from across the world, is an ideal medium for the purpose of Networked Musical Collaboration. Most conventional systems that enable networked musical collaboration minimize network latency by focusing on the transfer of auditory information at the expense of visual feedback. Studies into human perception have shown that sensory integration of audio and visual stimuli can take place even when there is a slight delay between the two signals. We studied the way changes in network latency effect participants’ auditory and visual perception in latency detection, latency tolerance and attention focus; in this paper, we explore the trade-off between the presence of AR visuals and the minimization of latency. Twenty-four participants were asked to play a hand drum and collaborate with a prerecorded remote musician rendered as an avatar in AR. Multiple trials involving different levels of audio-visual latency were conducted. We then analyzed the subjective responses of the participants together with the recorded musical information from each session. Results indicate a minimum noticeable delay value–defined as the highest amount of delay that can be experienced before two stimulated senses are perceived as separate events–between 160 milliseconds (ms) and 320 ms. Players also reported that a delay between sound and an accompanying avatar animation became less tolerable at 320 ms of delay, but was never completely intolerable, even up to 1200 ms of delay. We conclude that players begin to notice delay at about 320 ms and most players can tolerate large delays between sound and animation.