We present Editing Reality, a mixed reality system that enables in-situ co-creation with generative AI directly within physical environments. Rather than treating generation as a one-shot command, the system supports embodied and iterative creation through speech, sketching, and direct manipulation, allowing users to generate, modify, erase, and retexture real-anchored virtual and reconstructed scene elements in place. Using a Research Through Design approach, we investigate how co-creation unfolds through iterative system development, a formative workshop, and expert review. From this process, we articulate a set of designerly framings that characterize in-situ co-creation as a negotiated, spatial, and temporal practice shaped by previews, accumulation, waiting, embodied evaluation, and learning the system as a spatial actor. We instantiate these ideas in a working system and report expert feedback highlighting both its creative potential and its design implications. Our work contributes a conceptual lens for understanding generative AI in mixed reality not as a one-shot automation tool, but as part of an embodied, situated creative process.

Mobile video calls are widely used to share information about real-world objects and environments with remote collaborators. While these calls provide valuable visual context in real time, the experience of interacting with people and moving around a space is significantly reduced when compared to co-located conversations. Recent work has demonstrated the potential of Mobile Augmented Reality applications to enable more spatial forms of collaboration across distance. To better understand the dynamics of mobile AR collaboration and how this medium compares against the status quo, we conducted a comparative structured observation study to analyze people’s perception of space and interaction with remote collaborators across mobile video calls and AR-based calls. Fourteen pairs of participants completed a spatial collaboration task using each medium. Through a mixed-methods analysis of session videos, transcripts, motion logs, post-task exercises, and interviews, we highlight how the choice of medium influences the roles and responsibilities that collaborators take on and the construction of a shared language for coordination. We discuss the importance of spatial reasoning with one’s body, how video calls help participants “be on the same page” more directly, and how AR calls enable both onsite and remote collaborators to engage with the space and each other in ways that resemble in-person interaction. Our study offers a nuanced view of the benefits and limitations of both mediums, and we conclude with a discussion of design implications for future systems that integrate mobile video and AR to better support spatial collaboration in its many forms.

Virtual reality (VR) offers benefits such as enhanced interest, engagement, and vocabulary retention when learning another language. Due to the immersive nature of VR, users can enter a virtual setting that provides context clues as to what a particular vocabulary word in the target language represents. Users can also interact with the object to help increase the potential of being able to retain the term in the target language for the object through association with its surroundings. While the influence of using contextually-relevant virtual environments and providing object interaction have been studied separately, the aim of this research is to identify whether the combination of object interaction and a context-relevant environment is acceptable to users and effective at promoting short-term retention for basic vocabulary related to everyday objects.

We investigate the role of doppelgängers on foreign language anxiety, trustworthiness, and Spanish vocabulary recall. Participants (n = 31) completed three Spanish language lessons presented by a doppelgänger, a generic VH, or a disembodied voice in immersive or desktop VR; completed the Foreign Language Anxiety Classroom Scale (FLACS), Ohanian inventory on trustworthiness; and were assessed on vocabulary recall. Our findings suggest potential avenues for leverage with the design of VHs for foreign language learning.

In-person human interaction relies on our spatial perception of each other and our surroundings. Current remote communication tools partially address each of these aspects. Video calls convey real user representations but without spatial interactions. Augmented and Virtual Reality (AR/VR) experiences are immersive and spatial but often use virtual environments and characters instead of real-life representations. Bridging these gaps, we introduce DualStream, a system for synchronous mobile AR remote communication that captures, streams, and displays spatial representations of users and their surroundings. DualStream supports transitions between user and environment representations with different levels of visuospatial fidelity, as well as the creation of persistent shared spaces using environment snapshots. We demonstrate how DualStream can enable spatial communication in real-world contexts, and support the creation of blended spaces for collaboration. A formative evaluation of DualStream revealed that users valued the ability to interact spatially and move between representations, and could see DualStream fitting into their own remote communication practices in the near future. Drawing from these findings, we discuss new opportunities for designing more widely accessible spatial communication tools, centered around the mobile phone.

Tinycade is a platform designed to help game designers build their own mini arcade games by hand. With this platform, one can craft functioning game controllers out of everyday materials such as cardboard and toothpicks. By utilizing computer vision markers, we can create a variety of inputs without a single wire. In this pictorial, we discuss the functionality of Tinycade and showcase three games that demonstrate the variety of controls possible with this platform.

Augmented Reality (AR), with its ability to make people feel like they are in the same space as friends from across the world, is an ideal medium for the purpose of Networked Musical Collaboration. Most conventional systems that enable networked musical collaboration minimize network latency by focusing on the transfer of auditory information at the expense of visual feedback. Studies into human perception have shown that sensory integration of audio and visual stimuli can take place even when there is a slight delay between the two signals. We studied the way changes in network latency effect participants’ auditory and visual perception in latency detection, latency tolerance and attention focus; in this paper, we explore the trade-off between the presence of AR visuals and the minimization of latency. Twenty-four participants were asked to play a hand drum and collaborate with a prerecorded remote musician rendered as an avatar in AR. Multiple trials involving different levels of audio-visual latency were conducted. We then analyzed the subjective responses of the participants together with the recorded musical information from each session. Results indicate a minimum noticeable delay value–defined as the highest amount of delay that can be experienced before two stimulated senses are perceived as separate events–between 160 milliseconds (ms) and 320 ms. Players also reported that a delay between sound and an accompanying avatar animation became less tolerable at 320 ms of delay, but was never completely intolerable, even up to 1200 ms of delay. We conclude that players begin to notice delay at about 320 ms and most players can tolerate large delays between sound and animation.

Contextual illusions, such as the Ebbinghaus Illusion, can be potentially used to improve or hinder reach-to-grasp interaction in a virtual environment by affecting the perception of object size and the action. However, the illusion effect has only been evaluated using 2D objects like discs or annuluses, and limited research has been conducted in a virtual environment. Moreover, it remains unknown how the sudden, or dynamic, change of surrounding features will impact the perception and then the action towards the object. In this paper, we conducted a series of experiments to evaluate the effects of 3D Ebbinghaus illusion with dynamic surrounding features on the task of reaching to grasp a 3D object in an immersive virtual environment. An innovative 3D perceptual judgment task revealed that the static 3D illusion affected the perceived size of the 3D object. Then, we experimentally manipulated the visual gain and loss of the 3D contextual inducers, the participant's virtual hand, and the entire 3D contextual object. Results revealed that the depth error (error in depth of reaching action) was influenced by a dynamic change in the size of the inducers, the fine adjustment of grasp was dependent on the visual presence of the virtual hand and vision of 3D contextual object was required for the reaching and grasping movements. These results will benefit the understanding of reach-to-grasp interactions in immersive virtual environments and can improve interaction design.

In this paper, we present our design of a high school summer course which uses our Visual Design Problem-based Learning Pedagogy using Virtual Environments as a strategy to teach computer science. Students solved visual design problems by creating 3D sculptures in an online virtual environment. These creations were further explored and refined in immersive display systems fostering embodied learning and remote peer presence and support. To achieve the desired design, students use programming and computing concepts, such as loops, to solve those visual design centered problems, i.e. solving for composition, positive/negative space, balance, as opposed to computational problems first, i.e. create a loop, a fractal, randomized lines, etc. We present results from a study conducted on three high school summer courses. We compared the use of our Visual Design Problem-based teaching strategy (students wrote code to solve challenges based on art and design principles) to a traditional strategy (students wrote code to demonstrate comprehension of computer science concepts). Our results showed that test scores were higher for students in our Visual Design Problem-based courses. This work may have a positive impact on computer science education by increasing engagement, knowledge acquisition, and self-directed learning.

In the real world, prior research in the field of perception and action has shown that individuals visually perceive objects differently based on their actual performance of an action on those objects, especially for sporting activities. For example, a golfer who performs well on putting a ball into the hole, will perceive that ball as larger (easier to hit) and the hole as larger (easier to put the ball in). We asked the following research question, can manipulation of visual perception of objects influence actual performance in the real world? Virtual objects are easily manipulated in Virtual Reality environments, therefore we investigated the use of Virtual Reality training where the properties of objects, such as size, were manipulated to influence perception on a golf putting task. In this paper, we present the results of our experimental user study. Putting performance increased after virtual reality training exposure when virtual objects were larger (perceived as easier to hit) and decreased when virtual objects were smaller (more difficult to hit). Our research has the potential to broaden the study of how virtual reality training can be used to improve sports training in a unique way.

This paper presents a novel interaction technique to combine multiple inputs and output modalities with a 3D printed open disc which is tracked and augmented with a virtual information. Props have been used in virtual and augmented environments to provide tangible associations to virtual elements. When used in a larger immersive environment or where a user is walking around, users may not have the capability to set these objects/props down or are limited to the number of hands, so users are reduced to the capability of using only two props or controllers. In our technique, users can instead use their body to store and organize the virtual objects associated with tangible props by physically hanging them on or attaching them to the body. This type of interaction is well-suited for both immersive visualizations and immersive virtual environments. In this paper, we present the results of an experimental evaluation of our Body Prop Interaction technique for each type of immersive environment and corresponding task type, demonstrating more effective usage in each scenario than a joystick or gesture-based interaction.

Prior work has shown that visual cues are more dominant than tactile cues for static objects. When texture and material properties are added, the reverse is true. However, it is not well understood which cues play a role in perceiving these textures when objects are moving. In this paper, we present an evaluation investigating speed and angular direction mismatch between visual and tactile information while objects are moving. Our results show that when objects are stationary or moving slowly, the sense of touch dominates visual cues; but as the speed increases, visual cues dominate tactile cues. When tactile and visual cues were not matching for moving virtual objects, visual cues were more dominant when the virtual objects moved at angles less than 30 degrees different than the tactile feedback in either direction. Therefore, potentially a 1D directional haptic device with physical texture substitutions may be able to convey sufficient realism when touching virtual objects that are moving faster than 4-8cm/s and with a difference in angular direction of 30 degrees or less. In this paper, we also present the limitations of this work and outline a research agenda for future investigation of visual and tactile cue dominance for perceiving textures of virtual objects moving in a virtual environment.

Immersive Visualizations of 3-dimensional scientific data pose unique interaction challenges. One challenge is to be able to interact with the visual volumetric representations of data using direct manipulation in a way that facilitates exploration and discovery, yet maintains data relationships. In this paper, we present Body-Prop Interaction, a novel tangible multimodal interface for immersive visualizations. Our interaction technique combines multiple input and output modalities with 3D printed open discs that are tracked and augmented with virtual information. To demonstrate our technique, we implemented interactive tasks for a volume visualization of graphite billet data. While demonstrated for this type of data visualization, our novel interaction technique has the potential to be used to interact with other types of augmented and virtual reality.

There are benefits to visualizing and exploring scientific data in immersive display systems. Interactive data analytics further exploit these benefits. Direct interaction techniques enable scientists to leverage their visual-cognitive skills for identifying and selecting visually rendered data. However, fatigue, data density, rendering, and visual encoding issues may result in reduced usage of immersive scientific visualizations. In this paper, we present algorithmic details of a novel volume selection interaction, implementation details, and discuss use case scenario. Our technique seeks to improve usage of immersive visualizations for scientific discovery and insight.

3D applications appear in every corner of life in the current technology era. There is a need for an ubiquitous 3D input device that works with many different platforms, from head-mounted displays (HMDs) to mobile touch devices, 3DTVs, and even the Cave Automatic Virtual Environments. We present 3DTouch, a novel wearable 3D input device worn on the fingertip for 3D manipulation tasks. 3DTouch is designed to fill the missing gap of a 3D input device that is self-contained, mobile, and universally works across various 3D platforms. This paper presents a low-cost solution to designing and implementing such a device. Our approach relies on a relative positioning technique using an optical laser sensor and a 9-DOF inertial measurement unit. The device employs touch input for the benefits of passive haptic feedback, and movement stability. On the other hand, with touch interaction, 3DTouch is conceptually less fatiguing to use over many hours than 3D spatial input devices. We propose a set of 3D interaction techniques including selection, translation, and rotation using 3DTouch. An evaluation also demonstrates the device's tracking accuracy of 1.10 mm and 2.33 degrees for subtle touch interaction in 3D space. We envision that modular solutions like 3DTouch opens up a whole new design space for interaction techniques to further develop on. With 3DTouch, we attempt to bring 3D applications a step closer to users.

3D applications appear in every corner of life in the current technology era. There is a need for an ubiquitous 3D input device that works with many different platforms, from head-mounted displays (HMDs) to mobile touch devices, 3DTVs, and even the Cave Automatic Virtual Environments. We present 3DTouch [1], a novel wearable 3D input device worn on the fingertip for 3D manipulation tasks. 3DTouch is designed to fill the missing gap of a 3D input device that is self-contained, mobile, and universally works across various 3D platforms. This video presents a working prototype of our solution, which is described in details in the paper [1]. Our approach relies on a relative positioning technique using an optical laser sensor (OPS) and a 9-DOF inertial measurement unit (IMU). The device employs touch input for the benefits of passive haptic feedback, and movement stability. On the other hand, with touch interaction, 3DTouch is conceptually less fatiguing to use over many hours than 3D spatial input devices. We propose a set of 3D interaction techniques including selection, translation, and rotation using 3DTouch. An evaluation also demonstrates the device's tracking accuracy of 1.10 mm and 2.33 degrees for subtle touch interaction in 3D space. We envision that modular solutions like 3DTouch opens up a whole new design space for interaction techniques to further develop on.

Our research is focused on the initial exploration of training a Brain Computer Interface (BCI) by using audio cues to navigate a virtual environment, instead of motor imagery. We have designed our BCI training and navigation to use audio cues that adhere to the dichotic listening (DL) mechanism so that users have an active choice for interaction or giving commands. We have implemented our Dichotic Listening BCI in a Virtual Environment so that it can be used to train users to apply those skills for a BCI to control a real-world assisted locomotive device or to simply navigate within the virtual environment. We hypothesized that the lateralization of the brain's response to music and speech will enhance the classification of a BCI. Unlike previous attempts in using the oddball paradigm, our results show that audio cues can be used simultaneously to elicit distinct EEG signals for BCI while still enabling an active choice for the user. We evaluated users' performance to actively input navigation tasks. Dichotic Listening BCI performs slightly better than Motor Imagery based BCI.

We report a series of experiments conducted to investigate the effects of travel technique on information gathering and cognition in complex virtual environments. In the first experiment, participants completed a non-branching multilevel 3D maze at their own pace using either real walking or one of two virtual travel techniques. In the second experiment, we constructed a real-world maze with branching pathways and modeled an identical virtual environment. Participants explored either the real or virtual maze for a predetermined amount of time using real walking or a virtual travel technique. Our results across experiments suggest that for complex environments requiring a large number of turns, virtual travel is an acceptable substitute for real walking if the goal of the application involves learning or reasoning based on information presented in the virtual world. However, for applications that require fast, efficient navigation or travel that closely resembles real-world behavior, real walking has advantages over common joystick-based virtual travel techniques.

We report on a study in which we investigate the effects of travel technique on simulator sickness in a real and virtual environment. Participants explored either a real maze or a virtual maze using either natural walking or simulated walking. Reported scores for measures of overall simulator sickness, disorientation, nausea, and occulomotor discomfort were all higher in the natural walking condition than either the simulated walking or real world conditions. This indicates that simulated walking is a better choice for reducing simulator sickness during tasks requiring a navigationally complex environment and a long amount of time.

Do human-human social interactions carry over to human-virtual human social interactions? How does this affect future interface designers? We replicated classical tests of social influence known as the social facilitation and inhibition effects. Social facilitation/inhibition theory states that when in the presence of others, people perform simple tasks better and complex tasks worse. Participants were randomly assigned to perform both simple and complex tasks alone and in the presence of either a real human, a projected virtual human, or a virtual human in a head-mounted display. Our results showed participants were inhibited by the presence of others, whether real or virtual. That is, participants performed worse on the complex task, both in terms of percent correct and reaction times, when in the presence of others than when alone. Social facilitation did not occur with the real or virtual human. We discuss these results and their implications for future interface designers.

Traditionally, students study plays by reading from a book. However, reading dialogue on paper does not always communicate the various emotions and actions that help people understand the significance of the person-to-person interactions that are represented.

Using both hands for 3D interaction allows users to transfer ingrained interaction skills, significantly increase performance on certain tasks, and reduce training [Bowman et al. 2005]. Guiard's framework of Bimanual manipulation states that different classes of bimanual actions exist [1997]. The Bimanual Asymmetric classification consists of both hands, performing different actions, coordinated to accomplish the same task. The Bimanual Symmetric classification involves each hand performing identical actions, either synchronously or asynchronously. Latulipe et al. compared a symmetric, dual-mouse technique for manipulation of spline curves, to two asymmetric dual-mouse techniques and a standard single-mouse technique. The symmetric technique performed best and was most preferred by participants [2006].

We describe a between-subjects experiment that compared four different methods of travel and their effect on cognition and paths taken in an immersive virtual environment (IVE). Participants answered a set of questions based on Crook's condensation of Bloom's taxonomy that assessed their cognition of the IVE with respect to knowledge, understanding and application, and higher mental processes. Participants also drew a sketch map of the IVE and the objects within it. The users' sense of presence was measured using the Steed-Usoh-Slater Presence Questionnaire. The participants' position and head orientation were automatically logged during their exposure to the virtual environment. These logs were later used to create visualizations of the paths taken. Path analysis, such as exploring the overlaid path visualizations and dwell data information, revealed further differences among the travel techniques. Our results suggest that, for applications where problem solving and evaluation of information is important or where opportunity to train is minimal, then having a large tracked space so that the participant can walk around the virtual environment provides benefits over common virtual travel techniques.