Virtual assistants (VAs) are increasingly positioned not just as tools, but as potential social companions—capable of offering either emotional or informational support. Yet, how these forms of support should adapt to varying task difficulties and embodiment styles remains underexplored. We conducted two user studies with cognitive and physical tasks to investigate how support type (emotional vs. informational) shapes user perceptions across variations in task difficulty (easy vs. hard) and embodiment (non-embodied vs. embodied). In Study 1, emotional support positively influenced users’ impressions of VA in easy tasks, while informational support was more effective in difficult tasks. In Study 2, participants also preferred emotional support for easy tasks, but differences between support types were less pronounced for difficult tasks. Notably, embodiment exerted no significant influence in either study. These findings underscore the role of context in shaping effective support strategies, offering design insights for VAs as social companions.

Physical augmentation technologies can extend human abilities beyond biological limitations, creating "superpower" experiences. However, as these technologies integrate with the human body, they can also introduce unfortunate negative effects due to the body’s constraints. Inspired by biology, especially the fly-catching mechanism of the flytrap plant, we designed "Flytrap Hand", a provocative artifact that grants users the superpower of accelerated grasping through a distance sensor and electrical muscle stimulation, while resulting in uncertainty. A mixed-method study (N = 12) revealed that participants appreciated the fortunate superpower effects, but also highlighted the unfortunate effects of discomfort, skepticism toward bodily automation, and ethical concerns. Building on these findings, we propose a design framework for designing more thoughtful superpower experiences that account for potential negative effects. Ultimately, with our work, we aim to help designers consider not only superpowers’ positive but also negative effects early on as they may be difficult to rectify in hindsight.

Designing superpowers in Human-Computer Interaction (HCI), often inspired by science fiction, has garnered increased attention. However, it is important to ask whether such superpower designs might have inherent negative side effects, especially considering that technological advances allow going beyond short demos to integrate these superpowers into everyday life. To understand the positive and negative side effects of superpower design, we created "EmoPals"and studied it in everyday life. EmoPals is a novel system inspired by telepathy, where one user's emotions are detected through a brain-computer interface and replicated on the other user's face through electrical muscle stimulation, therefore one user's happiness makes the other smile and vice versa. A 5-day field study with 12 participants suggests that EmoPals can strengthen emotional connections and facilitate empathy, however, it also highlights the negative side effects of amplifying negative emotions and social discomfort. We propose five design recommendations for designing superpowers that account for negative side effects. Ultimately, we aim to deepen our understanding of superpower design for everyday life.

We introduce bichronous collaboration in an eXtended Reality (XR) remote collaboration workflow that seamlessly links asynchronous review with synchronous cooperation on physical tasks. The aim is to reduce the misunderstandings and errors that often arise during remote work. The system records a novice's actions and presents them with three spatiotemporal cues-2D, 3D Basic, and 3D Action. An expert can review and provide feedback asynchronously, and also guide the novice synchronously to fix the error. A user study showed that the 3D Action cue allowed experts to identify faults more quickly, enabled novices to resolve problems in less time with higher success rates than a conventional 2D timeline, without increasing cognitive workload or cybersickness. These findings highlight the value of coupling asynchronous insight with synchronous teamwork and position bichronous collaboration as a practical solution for remote troubleshooting, training, and inspection in XR environments.

People often attempt to mask their true emotions through deliberate facial expressions, but such efforts are not always successful. In contrast, emotional concealment could be more easily achieved in Virtual Reality (VR) when appropriate functionalities are available. This study introduces three techniques in VR that enable users to manually adjust emotional facial expression while still reflecting real-time facial tracking results. The Ekman (Ek) technique allows users to select six discrete emotions via button interaction, while the Scrollable-Ekman (SEk) technique extends this by allowing users to scale the intensity of the selected emotion. The Arousal-Valence (AV) technique offers nuanced control within a two-dimensional arousal-valence space. We evaluated these techniques against a baseline condition that synchronizes users' natural facial expressions, focusing on the expression of happiness, sadness, and anger. In most measurements, the Ek and SEk showed better results compared to the baseline and the AV techniques. Notably, the SEk technique was particularly effective in enhancing hedonic quality. During the free-flowing conversation, the most critical factor was the timely and well-synchronized coordination between speech and controlled facial expressions. Participant satisfaction also varied by usage style: those who tried to use the techniques continuously and naturally to mimic real-life communication reported lower satisfaction, while those who used them occasionally for playful or exaggerated expressions tended to report higher satisfaction.

Managing negative emotions through emotion regulation (ER) is key to mental well-being. While virtual reality (VR) shows promise for supporting ER, prior work has primarily focused on selfregulation. This paper introduces a virtual agent that helps users manage emotions through conversation-based ER strategies. We compared three conditions: no agent, an agent with non-supportive responses, and an agent with ER-supportive responses. Results showed that the ER-supportive agent significantly improved users' emotional states and overall experience. Building on this, we conducted a second experiment to examine how the agent's appearance (realistic vs. cartoon) and voice tone (emotional vs. neutral) affect ER. Results indicated that an emotional voice tone improved users' ability to regulate emotions. Although a realistic appearance did not directly improve ER, it increased users' trust and sense of social presence. This paper contributes to VR and human-agent interaction by demonstrating the potential of virtual agents to support ER and offering design implications for future ER-supportive agents.

Empathy is central to social interaction, yet how it is externally expressed in virtual reality (VR) communication remains underexplored. In this study, we examined how directionality-aware cues of empathy, such as mimicry, eye contact, and body proximity, relate to cognitive and emotional empathy. We designed high- and low-empathy scenarios and recruited participants with acting experience to ensure clear emotional expressions. Our findings indicate that facial mimicry patterns differ by empathy type: cognitive empathy involves subtle, speech-related muscle movements, whereas emotional empathy is associated with more intense affective expressions. Interestingly, we also found that while facial expressions and lower-body mimicry tend to emerge unconsciously, upper-body mimicry occurs more consciously, suggesting distinct pathways of empathic embodiment. We also observed that vocal intensity mimicry and pitch variability serve as important indicators of empathy, and a consistent hand approach is closely linked to empathy. Additionally, emotional empathy fosters longer eye contact, whereas cognitive empathy stabilizes gaze and head movements. Finally, we constructed machine learning models to predict empathy from these external expressions. Our best classifier achieved an accuracy of 0.756 for cognitive empathy and 0.704 for emotional empathy, indicating the feasibility of objective assessment. These findings provide a deeper understanding of how empathy is manifested in VR communication and support the development of empathy-aware virtual agents and training systems.

Augmented Reality (AR) has demonstrated significant potential in task guidance by offering hands-free, context-sensitive instructions. However, effective AR-based task guidance requires maintaining user focus, as lapses in attention can lead to inefficiencies and errors. This paper describes a novel approach for integrating real-time gaze and location tracking to dynamically adapt AR task guidance, adjusting the playback speed accordingly. A user study with 20 participants assessed the effectiveness of gaze-based and locationbased focus detection across four conditions: baseline, gaze-only, location-only, and combined interaction. The results indicate that both gaze-based and location-based guidance significantly reduce error rates and improve task efficiency, however, the simultaneous use of both methods leads to cognitive overload, increased errors, and interaction complexity. Participants found that the adaptive playback speed adjustments helped manage attention. These findings highlight the importance of balancing adaptive cues in AR task guidance to optimize user performance without overwhelming cognitive resources.

We introduce HeadZoom, a hands-free interaction technique for navigating two-dimensional visual content using head movements. HeadZoom enables fluid zooming and panning using only real-time head tracking. It supports natural control in applications such as map exploration, radiograph inspection, and image browsing, where physical interaction is limited. We evaluated HeadZoom in a withinsubjects study comparing three interaction techniques-Static, Tilt Zoom, and Parallel Zoom-across spatial, error, and subjective metrics. Parallel Zoom significantly reduced total head movement compared to Static and Tilt modes. Users reported significantly lower perceived exertion for Parallel Zoom, confirming its suitability for prolonged or precision-based tasks. By minimizing movement demands while maintaining task effectiveness, HeadZoom advances the design of head-based 2D interaction in VR and creates new opportunities for accessible hands-free systems for image exploration.

Co-speech gestures are crucial for enriching both human-human and human-agent communications. Yet, the specific impacts of gesture size—especially when being generated by advanced data-driven techniques—remain underexplored. This study investigates how varying gesture sizes affect human-agent interactions across two distinct persuasive contexts (informational and emotional), with a focus on social outcomes such as persuasion and empathy. We conducted two controlled experiments, each involving 36 participants, comparing three gesture conditions: Minimal gesture, Small gesture, and Large gesture conditions. Experiment 1, set in an informational sales context, showed that small and large gestures significantly enhanced persuasive effectiveness, social presence, and communication quality compared to the minimal gesture condition, although no meaningful differences emerged between small and large gestures. In contrast, Experiment 2, situated in an emotionally charged context, revealed that larger gestures progressively amplified both persuasive impact and perceived empathy. These findings highlight that gesture size matters in emotionally intensive communications and the substantial social benefits of deep-learning techniques for gesture generation.

Prior research has largely focused on AR or VR alone, but recent CR advances show the benefit of fluidly combining them. This article presents a novel cross-reality (CR) task guidance system designed to support asynchronous task guidance on spatial tasks. The system enables (1) recording and playback of user actions within large-scale workspaces and (2) seamless transitions between augmented reality (AR), world-in-miniature (WIM), and virtual reality (VR) interfaces. A user study demonstrated that the CR interface significantly improved task efficiency, reduced required physical activity, enhanced spatial memory recall, and lowered perceived task load. These findings highlight the system’s ability to bridge temporal and spatial gaps in asynchronous task guidance and suggest the broader potential of CR technologies for collaborative workflows. Limitations and design implications are also discussed to inform future research on CR interface development.

This paper introduces the Simultaneous Presence (SP) Continuum, a novel concept designed to enhance the use of portals in virtual reality (VR) by understanding users' experiences across multiple environments. Portals traditionally provide access to secondary worlds, but their limited field of view (FoV) can hinder full engagement with these environments. To overcome this, we introduce portal overlays, which expand the portal's FoV by superimposing the secondary world over the user's view of the primary world. We explore several overlay techniques-Contours, Blended (Opacity and Stencil), and Absolute-to adjust user experience across the SP Continuum. The Contours overlay, renders only the edges of objects, offering a minimalistic view and immersion of the secondary world. The Blended overlay allows for adjustable immersion between worlds through opacity or the distribution of pixels. The Absolute overlay virtually immerses users completely in the secondary world. Importantly, these overlays are context-activated instead of always on, to suit situational needs. Our study revealed high SP was possible in both worlds, and overlays significantly impacted users' experiences on the SP Continuum. The Blended overlay provided balanced SP, while Contours had the highest primary presence but lower secondary presence. In contrast, the Absolute overlay, alternating full immersion between worlds, had the highest secondary presence but resulted in reduced primary presence, slower navigation and selection times, higher effort and frustration, and lower usability.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 24th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2025), which will be held between October 8 and 12, 2025, in Daejeon, South Korea. ISMAR continues the over twenty-year-long tradition of IWAR, ISMR, and ISAR, and is the world's premier conference for Mixed and Augmented Reality.

Virtual reality (VR) is increasingly adopted for collaborative teaching and learning, enabling immersive and interactive experiences. As Artificial Intelligence (AI) tutors begin to take on roles alongside human instructors, it becomes crucial to understand how their integration influences interaction dynamics and role perception in these settings. This study investigates the role of Embodied Virtual Agents (EVAs) substituting human instructors in virtual training, specifically addressing the previously underexplored issue of EVA appearance consistency with instructors in Human-Agent Teaming (HAT). We recruited 21 participants to compare three conditions: No Agent, Shared-Appearance (SA) EVA, and Unique-Appearance (UA) EVA, where an EVA substitutes for the instructor during temporary absences. We evaluated collaboration efficiency, user perception/preference, and HAT dynamics. Our findings confirm that EVAs significantly enhance task efficiency compared to no support and reveal a key trade-off regarding appearance: SA fosters perceived continuity and trust but risks ambiguity and uncanny effects, while UA provides transparency and role clarity but may disrupt experiential coherence. These results have implications for designing dynamic HAT systems where control may shift. We discuss the benefits and limitations of each approach and offer design recommendations for future mixed-agency interfaces.

In this paper, we introduce a novel Virtual Reality (VR) navigation method using gaze ray and hand, named RayHand navigation. It supports controlling navigation speed and direction by quickly indicating the initial direction using gaze and then using dexterous hand movement for controlling the speed and direction based on the relative position between the gaze ray and user's hand. We conducted a user study comparing our approach to the head-hand and torso-leaning-based navigation methods, and also evaluated their learning effect. The results showed that the RayHand and head-hand navigations were less physically demanding than the torso-leaning navigation, and the RayHand supported rich navigation experience with high hedonic quality and solved the issue of the user unintentionally stepping out from the designated interaction area. In addition, our approach showed a significant improvement over time with a learning effect.

This paper introduces foundational techniques for Adaptive Portals (AP); portals that adjust to users’ needs in Virtual Reality (VR). At the forefront of our contributions is the AP-Reach technique, with Entry and Exit strategies, designed to overcome the spatial limitations of portals and VR, enabling users to comfortably extend their reach to objects beyond their immediate vicinity. In parallel, we present PH-Reach as an alternative to the AP concept, also aimed at extending user reach inside portals. The effectiveness of these reach techniques is evaluated through a comparative user study, which compared Entry AP-Reach, Exit AP-Reach, and PHReach techniquesagainst traditional natural reach as a baseline. Results show Exit AP-Reach surpassed both its counterparts and the baseline to various degrees in terms of reducing task completion time, physical movement/demand, effort, and frustration. Additionally, this paper introduces AP-Bounds and AP-Restriction.

Representing emotion in collaborative Virtual Reality (VR) environments is an emerging topic, as VR can enable humans to express augmented emotions beyond their normal abilities. This research explores how emotion can be represented in collaborative VR beyond facial expressions. We developed a virtual system that communicates emotion using three sensory modalities (textual, auditory and visual) through two spatiotemporal representations (human-form avatar and superpower). We show real-time emotion through a natural avatar and objectify emotion states into superpower phenomena in in-situ environments for time periods. We incorporated subjective, physiological and behavioural measures to evaluate emotion in an asynchronous VR collaboration scenario. The results suggested that showing emotions through the avatar and superpower augmentations (audio-visual) provided the best immersive VR experience, where users were more aroused, and the positive emotion felt more dominating. We also found that understanding emotion requires easy and relatable visuals that people commonly acknowledge, whereas arousing emotion requires a change of environmental contexts to indicate different states. We provide design insights for using multisensory modalities in empathic VR systems to address the lack of a standardised representation of emotion in collaborative VR.

Technology-facilitated “superpowers” are a popular topic in human-computer interaction, providing people with the experience of having superhero-like fantastic abilities. However, science fiction literature tells us that superpowers can also be unfortunate. We designed Wi-Fi Twinge, a system that uses electrical muscle stimulation to extend human perception to sense unseen Wi-Fi signals, giving people an allergy-like twinge reaction to Wi-Fi: a superpower that can be seen as unfortunate while also having potential benefits. Through an in-the-wild study, interviews with 12 participants revealed that although the superpower from Wi-Fi Twinge induced negative feelings and impacted certain day-to-day activities, it improved awareness of self and the surrounding environment. We used these results to offer design tactics for creating future superpower systems that consider potential unfortunate side effects. Ultimately, we aim to advance our understanding of technology-facilitated superpowers.

This research investigates fast and precise Virtual Reality (VR) sketching methods with different tool-based asymmetric interfaces. In traditional real-world drawing, artists commonly employ an asymmetric interaction system where each hand holds different tools, facilitating diverse and nuanced artistic expressions. However, in virtual reality (VR), users are typically limited to using identical tools in both hands for drawing. To bridge this gap, we aim to introduce specifically designed tools in VR that replicate the varied tool configurations found in the real world. Hence, we developed a VR sketching system supporting three hybrid input techniques using a standard VR controller, a VR stylus, or a data glove. We conducted a formal user study consisting of an internal comparative experiment with four conditions and three tasks to compare three asymmetric input methods with each other and with a traditional symmetric controller-based solution based on questionnaires and performance evaluations. The results showed that in contrast to symmetric dual VR controller interfaces, the asymmetric input with gestures significantly reduced task completion times while maintaining good usability and input accuracy with a low task workload. This shows the value of asymmetric input methods for VR sketching. We also found that the overall user experience could be further improved by optimizing the tracking stability of the data glove and the VR stylus.

This paper introduces Point & Portal, a novel Virtual Reality (VR) interaction technique, inspired by Point & Teleport. This new technique enables users to configure portals using pointing actions, and supports seamless action at a distance and navigation without requiring line of sight. By supporting multiple portals, Point & Portalenables users to create dynamic portal configurations to manage multiple remote tasks. Additionally, this paper introduces Relative Portal Positioning for reliable portal interactions, and the concept of maintaining Level Portals. In a comparative user study, Point & Portal demonstrated significant advantages over the traditional Point & Teleport technique to bring interaction devices within-arm’s reach. In the presence of obstacles, Point & Portal exhibited faster speed, lower cognitive load and was preferred by participants. Overall, participants required less physical movement, pointing actions, and reported higher involvement and “good” usability.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 22nd IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2023), which will be held as a hybrid conference between October 16 and 20, 2023 in Sydney, Australia. ISMAR continues the over twenty-year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

In this paper, we explore how virtual replicas can enhance Mixed Reality (MR) remote collaboration with a 3D reconstruction of the task space. People in different locations may need to work together remotely on complicated tasks. For example, a local user could follow a remote expert's instructions to complete a physical task. However, it could be challenging for the local user to fully understand the remote expert's intentions without effective spatial referencing and action demonstration. In this research, we investigate how virtual replicas can work as a spatial communication cue to improve MR remote collaboration. This approach segments the foreground manipulable objects in the local environment and creates corresponding virtual replicas of physical task objects. The remote user can then manipulate these virtual replicas to explain the task and guide their partner. This enables the local user to rapidly and accurately understand the remote expert's intentions and instructions. Our user study with an object assembly task found that using virtual replica manipulation was more efficient than using 3D annotation drawing in an MR remote collaboration scenario. We report and discuss the findings and limitations of our system and study, and present directions for future research.

Transformative portals are a novel technique for supporting visualizations and interactions between locations that are not co-located in virtual reality (VR). We are interested in improving upon current portal rendering for VR, and supporting VR users to create new portals quickly and accurately. In this paper, we introduce a new high-level algorithm for rendering transformative portals in VR and present the benefits of our approach. By leveraging single-pass stereo and stencil portal rendering, we developed our algorithm to support multiple portals with “infinite” recursion, while running performant on mobile VR devices. We then benchmarked our performance against other common portal rendering implementations. In addition, we focused our research on VR interactions for handheld portals, in which the initial placement of the portal’s destination is imperative to its effectiveness. In this paper, we present four new portal creation interactions for handheld portals: fishing reel, raycast with fishing reel, marker, and projectile curve. Based on existing research, we also established standard controls for fine-grained portal manipulation. In a user study, we compared and evaluated how well the creation interactions performed and argue that projectile curve is the most suitable general-purpose technique.

In this paper, we share real-time collaborative gaze behaviours, hand pointing, gesturing, and heart rate visualisations between remote collaborators using a live 360 ° panoramic-video based Mixed Reality (MR) system. We first ran a pilot study to explore visual designs to combine communication cues with biofeedback (heart rate), aiming to understand user perceptions of empathic collaboration. We then conducted a formal study to investigate the effect of modality (Gaze+Hand, Hand-only) and interface (Near-Gaze, Embodied). The results show that the Gaze+Hand modality in a Near-Gaze interface is significantly better at reducing task load, improving co-presence, enhancing understanding and tightening collaborative behaviours compared to the conventional Embodied hand-only experience. Ranked as the most preferred condition, the Gaze+Hand in Near-Gaze condition is perceived to reduce the need for dividing attention to the collaborator’s physical location, although it feels slightly less natural compared to the embodied visualisations. In addition, the Gaze+Hand conditions also led to more joint attention and less hand pointing to align mutual understanding. Lastly, we provide a design guideline to summarize what we have learned from the studies on the representation between modality, interface, and biofeedback.

In this paper, we present a 360° panoramic Mixed Reality (MR) sys-tem that visualises shared gaze cues using contextual speech input to improve task coordination. We conducted two studies to evaluate the design of the MR gaze-speech interface exploring the combinations of visualisation style and context control level. Findings from the first study suggest that an explicit visual form that directly connects the collaborators’ shared gaze to the contextual conversation is preferred. The second study indicates that the gaze-speech modality shortens the coordination time to attend to the shared interest, making the communication more natural and the collaboration more effective. Qualitative feedback also suggest that having a constant joint gaze indicator provides a consistent bi-directional view while establishing a sense of co-presence during task collaboration. We discuss the implications for the design of collaborative MR systems and directions for future research.

In a remote collaboration involving a physical task, visualising gaze behaviours may compensate for other unavailable communication channels. In this paper, we report on a 360° panoramic Mixed Reality (MR) remote collaboration system that shares gaze behaviour visualisations between a local user in Augmented Reality and a remote collaborator in Virtual Reality. We conducted two user studies to evaluate the design of MR gaze interfaces and the effect of gaze behaviour (on/off) and gaze style (bi-/uni-directional). The results indicate that gaze visualisations amplify meaningful joint attention and improve co-presence compared to a no gaze condition. Gaze behaviour visualisations enable communication to be less verbally complex therefore lowering collaborators' cognitive load while improving mutual understanding. Users felt that bi-directional behaviour visualisation, showing both collaborator's gaze state, was the preferred condition since it enabled easy identification of shared interests and task progress.

Navigation is a primary interaction in virtual reality. Previous research has explored different forms of artificial locomotion techniques for navigation, including hand gestures and body motions. However, few studies have investigated force-based foot gestures as a locomotion technique. We present three force-based foot gestures (Foot Fly, Foot Step and Foot Teleportation) for navigation in a virtual environment, relying on surface electromyography sensors readings from leg muscles. A pilot study comparing our techniques with controller-based techniques indicates that force-based foot gestures can provide a fun and engaging alternative. Of all six input techniques evaluated, Foot Fly was often most preferred despite requiring more exertion than the Controller Fly technique.

In this paper, we present a novel collaboration tool, OmniGlobeVR, which is an asymmetric system that supports communication and collaboration between a VR user (occupant) and multiple non-VR users (designers) across the virtual and physical platform. OmniGlobeVR allows designer(s) to explore the VR space from any point of view using two view modes: a 360° first-person mode and a third-person mode. In addition, a shared gaze awareness cue is provided to further enhance communication between the occupant and the designer(s). Finally, the system has a face window feature that allows designer(s) to share their facial expressions and upper body view with the occupant for exchanging and expressing information using nonverbal cues. We conducted a user study to evaluate the OmniGlobeVR, comparing three conditions: (1) first-person mode with the face window, (2) first-person mode with a solid window, and (3) third-person mode with the face window. We found that the first-person mode with the face window required significantly less mental effort, and provided better spatial presence, usability, and understanding of the partner's focus. We discuss the design implications of these results and directions for future research.

This research investigates enhancing first-person view (FPV) task instruction videos by applying Augmented Reality (AR) visualisation of spatial cues. With personal mobile devices, recording and sharing a video clip has become very easy, and how-to videos are becoming popular on social video sharing services. Instructional videos are actively used not only in formal education and training, but also in everyday life. However, video clips are limited to two-dimensional representation of the task space, making it hard for the viewer to follow and match the objects in the video to those in the real world task space. We propose augmenting task instruction videos with AR visualisation of spatial cues to overcome this problem, focusing on creating and viewing FPV instruction videos. We designed and implemented a prototype system, AR Tips, which allows users to capture and share augmented FPV instruction videos on a wearable AR device. We conducted a user study to evaluate the benefit of our approach, and the results showed that with the help of augmented spatial cues users better understood the instructions, performed the tasks faster with fewer errors, and had lower mental effort.

We have proposed a proof of concept of a gaze window interface. A gaze window interface uses the user's gaze point to show the relevant content nearby the current gaze point. A system of a gaze window interface has been implemented that allows users to manage looking at multiple objects of interest on the screen while the system is responsive to the user's gaze points with relevant information shown in the gaze window. We also conducted user studies to investigate the effects of the interface on user performance and behaviors. In this paper, we report on a user study in which the gaze window was compared with a mouse window for supporting single user data entry tasks. We describe details of the study design and conduction and present the results.

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

We propose a multi-scale Mixed Reality (MR) collaboration between the Giant, a local Augmented Reality user, and the Miniature, a remote Virtual Reality user, in Giant-Miniature Collaboration (GMC). The Miniature is immersed in a 360-video shared by the Giant who can physically manipulate the Miniature through a tangible interface, a combined 360-camera with a 6 DOF tracker. We implemented a prototype system as a proof of concept and conducted a user study (n=24) comprising of four parts comparing: A) two types of virtual representations, B) three levels of Miniature control, C) three levels of 360-video view dependencies, and D) four 360-camera placement positions on the Giant. The results show users prefer a shoulder mounted camera view, while a view frustum with a complimentary avatar is a good visualization for the Miniature virtual representation. From the results, we give design recommendations and demonstrate an example Giant-Miniature Interaction.

Remote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D virtual reconstruction of their surroundings. However, each of these techniques has benefits and drawbacks. In this paper we explore mixing 360 video and 3D reconstruction together for remote collaboration, by preserving benefits of both systems while reducing drawbacks of each. We developed a hybrid prototype and conducted user study to compare benefits and problems of using 360 or 3D alone to clarify the needs for mixing the two, and also to evaluate the prototype system. We found participants performed significantly better on collaborative search tasks in 360 and felt higher social presence, yet 3D also showed potential to complement. Participant feedback collected after trying our hybrid system provided directions for improvement.

In this paper, we explore the effect of showing a remote partner close to user gaze point in a teleconferencing system. We implemented a gaze following function in a teleconferencing system and investigate if this improves the user's feeling of emotional interdependence. We developed a prototype system that shows a remote partner close to the user's current gaze point and conducted a user study comparing it to a condition displaying the partner fixed in the corner of a screen. Our results showed that showing a partner close to their gaze point helped users feel a higher level of emotional interdependence. In addition, we compared the effect of our method between small and big displays, but there was no significant difference in the users' feeling of emotional interdependence even though the big display was preferred.

Research has shown that Mixed Presence Groupware (MPG) systems are a valuable collaboration tool. However research into MPG systems is limited to a handful of tabletop and Virtual Reality (VR) systems with no exploration of Head-Mounted Display (HMD) based Augmented Reality (AR) solutions. We present a new system with two local users and one remote user using HMD based AR interfaces. Our system provides tools allowing users to layout a room with the help of a remote user. The remote user has access to a marker and pointer tools to assist in directing the local users. Feedback collected from several groups of users showed that our system is easy to learn but could have increased accuracy and consistency.

We propose enhancing live 360 panorama-based Mixed Reality (MR) remote collaboration through supporting visual annotation cues. Prior work on live 360 panorama-based collaboration used MR visualization to overlay visual cues, such as view frames and virtual hands, yet they were not registered onto the shared physical workspace, hence had limitations in accuracy for pointing or marking objects. Our prototype system uses spatial mapping and tracking feature of an Augmented Reality head-mounted display to show visual annotation cues accurately registered onto the physical environment. We describe the design and implementation details of our prototype system, and discuss on how such feature could help improve MR remote collaboration.

This paper investigates the effect of avatar appearance on Social Presence and users' perception in an Augmented Reality (AR) telep-resence system. Despite the development of various commercial 3D telepresence systems, there has been little evaluation and discussions about the appearance of the collaborator's avatars. We conducted two user studies comparing the effect of avatar appearances with three levels of body part visibility (head & hands, upper body, and whole body) and two different character styles (realistic and cartoon-like) on Social Presence while performing two different remote collaboration tasks. We found that a realistic whole body avatar was perceived as being the best for remote collaboration, but an upper body or cartoon style could be considered as a substitute depending on the collaboration context. We discuss these results and suggest guidelines for designing future avatar-mediated AR remote collaboration systems.

Augmented Reality (AR) mirrors can show virtual objects overlaid onto the physical world reflected in the mirror. Optical-reflection type AR mirror displays use half-silvered mirrors attached in front of a digital display. However, prior work suffered from visual depth mismatch between the optical reflection of the 3D physical space and 2D images displayed on the surface of the mirror. In this research, we use 3D visualisation to overcome this problem and improve the user experience by providing better depth perception for watching and interacting with the content displayed on an AR mirror. As a proof of concept, we developed two prototype optical-reflection type 3D AR mirror displays, one using glasses-free multi-view 3D display and another using a head tracked 3D stereoscopic display that supports hand gesture interaction.

Mixed Reality (MR) remote collaboration provides an enhanced immersive experience where a remote user can provide verbal and nonverbal assistance to a local user to increase the efficiency and performance of the collaboration. This is usually achieved by sharing the local user's environment through live 360 video or a 3D scene, and using visual cues to gesture or point at real objects allowing for better understanding and collaborative task performance. While most of prior work used one of the methods to capture the surrounding environment, there may be situations where users have to choose between using 360 panoramas or 3D scene reconstruction to collaborate, as each have unique benefits and limitations. In this paper we designed a prototype system that combines 360 panoramas into a 3D scene to introduce a novel way for users to interact and collaborate with each other. We evaluated the prototype through a user study which compared the usability and performance of our proposed approach to live 360 video collaborative system, and we found that participants enjoyed using different ways to access the local user's environment although it took them longer time to learn to use our system. We also collected subjective feedback for future improvements and provide directions for future research.

Head and eye movement can be leveraged to improve the user's interaction repertoire for wearable displays. Head movements are deliberate and accurate, and provide the current state-of-the-art pointing technique. Eye gaze can potentially be faster and more ergonomic, but suffers from low accuracy due to calibration errors and drift of wearable eye-tracking sensors. This work investigates precise, multimodal selection techniques using head motion and eye gaze. A comparison of speed and pointing accuracy reveals the relative merits of each method, including the achievable target size for robust selection. We demonstrate and discuss example applications for augmented reality, including compact menus with deep structure, and a proof-of-concept method for on-line correction of calibration drift.

We present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user's gaze direction and body gestures while it transforms in size and orientation to stay within the AR user's field of view. A user study was conducted to evaluate Mini-Me in two collaborative scenarios: an asymmetric remote expert in VR assisting a local worker in AR, and a symmetric collaboration in urban planning. We found that the presence of the Mini-Me significantly improved Social Presence and the overall experience of MR collaboration.

Sharing and watching live 360 panorama video is available on modern social networking platforms, yet the communication is often a passive one-directional experience. This research investigates how to further improve live 360 panorama based remote collaborative experiences by adding Mixed Reality (MR) cues. SharedSphere is a wearable MR remote collaboration system that enriches a live captured immersive panorama based collaboration through MR visualisation of non-verbal communication cues (e.g., view awareness and gestures cues). We describe the design and implementation details of the prototype system, and report on a user study investigating how MR live panorama sharing affects the user's collaborative experience. The results showed that providing view independence through sharing live panorama enhances co-presence in collaboration, and the MR cues help users understanding each other. Based on the study results we discuss design implications and future research direction.

The advancements in Mixed Reality (MR), Unmanned Aerial Vehicle, and multi-scale collaborative virtual environments have led to new interface opportunities for remote collaboration. This paper explores a novel concept of flying telepresence for multi-scale mixed reality remote collaboration. This work could enable remote collaboration at a larger scale such as building construction. We conducted a user study with three experiments. The first experiment compared two interfaces, static and dynamic IPD, on simulator sickness and body size perception. The second experiment tested the user perception of a virtual object size under three levels of IPD and movement gain manipulation with a fixed eye height in a virtual environment having reduced or rich visual cues. Our last experiment investigated the participant's body size perception for two levels of manipulation of the IPDs and heights using stereo video footage to simulate a flying telepresence experience. The studies found that manipulating IPDs and eye height influenced the user's size perception. We present our findings and share the recommendations for designing a multi-scale MR flying telepresence interface.

Using game balancing techniques can provide the right level of challenge and hence enhance player engagement for sport players with different skill levels. Digital technology can support and enhance balancing techniques in sports, for example, by adjusting players' level of intensity based on their heart rate. However, there is limited knowledge on how to design such balancing and its impact on the user experience. To address this we created two novel balancing techniques enabled by digitally augmenting a table tennis table. We adjusted the more skilled player's performance by inducing two different styles of play and studied the effects on game balancing and player engagement. We showed that by altering the more skilled player's performance we can balance the game through: (i) encouraging game mistakes, and (ii) changing the style of play to one that is easier for the opponent to counteract. We outline the advantages and disadvantages of each approach, extending our understanding of game balancing design. We also show that digitally augmenting sports offers opportunities for novel balancing techniques while facilitating engaging experiences, guiding those interested in HCI and sports.

We present results from research exploring the effect of sharing virtual gaze and pointing cues in a wearable interface for remote collaboration. A local worker wears a Head-mounted Camera, Eye-tracking camera and a Head-Mounted Display and shares video and virtual gaze information with a remote helper. The remote helper can provide feedback using a virtual pointer on the live video view. The prototype system was evaluated with a formal user study. Comparing four conditions, (1) NONE (no cue), (2) POINTER, (3) EYE-TRACKER and (4) BOTH (both pointer and eye-tracker cues), we observed that the task completion performance was best in the BOTH condition with a significant difference of POINTER and EYETRACKER individually. The use of eye-tracking and a pointer also significantly improved the co-presence felt between the users. We discuss the implications of this research and the limitations of the developed system that could be improved in further work.

Playing exertion games with others can be engaging. However, players with different physical skill levels competing against each other can experience reduced engagement because they are either not challenged enough, or challenged too much. Balancing methods can address this; however, there is only limited understanding of balancing in exertion games. In this paper, we identify two distinct dimensional balancing techniques: "internal adjustment" and "external adjustment". We report results from a study where we measured player engagement after applying these adjustments to a digital table tennis game and the traditional table tennis game, finding two disengagement factors: "unexpected physical challenges" and "unacceptable competitive advantage". Based on these factors we derived a set of exertion game design considerations. We conclude that applying digital technology to a physical game can change the required skill level to play the game, and this can affect the impact of these adjustments on player engagement. These results enhances our understanding of balancing in exertion games, supporting the benefits of playing exertion games with others.

Video conferencing is becoming more widely used in areas other than face-to-face conversation, such as sharing real world experience with remote friends or family. In this paper we explore how adding augmented visual communication cues can improve the experience of sharing remote task space and collaborating together. We developed a prototype system that allows users to share live video view of their task space taken on a Head Mounted Display (HMD) or Handheld Display (HHD), and communicate through not only voice but also using augmented pointer or annotations drawn on the shared view. To explore the effect of having such an interface for remote collaboration, we conducted a user study comparing three video-conferencing conditions with different combination of communication cues: (1) voice only, (2) voice + pointer, and (3) voice + annotation. The participants used our remote collaboration system to share a parallel experience of puzzle solving in the user study, and we found that adding augmented visual cues significantly improved the sense of being together. The pointer was the most preferred additional cue by users for parallel experience, and there were different states of the users' behavior found in remote collaboration.

In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform. Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems. In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging. We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.

In this research we explore how a handheld display (HHD) can be used to provide input into an Augmented Reality (AR) conferencing application shown on a head mounted display (HMD). Although AR has successfully been used for many collaborative applications, there has been little research on using HHD and HMD together to enhance remote conferencing. This research investigates two different HHD interfaces and methods for supporting file sharing in an AR conferencing application. A formal evaluation compared four different conditions and found that an Exocentric view and using Visual cues for requesting content produced the best performance. The results were used to create a set of basic design guidelines for future research and application development.

Most mobile Augmented Reality (AR) systems use either a head mounted display (HMD) or a handheld display (HHD) as a hardware platform. As mobile devices become more affordable, it becomes more common that users own more than one mobile device and use them together. In this research, we investigate Hybrid AR systems that use both HMD and HHD for AR visualization and interaction. In addition to a simple approach of using HMD as a display and HHD as an input device (e.g. a touch pad or a pointer), we further explore novel interaction techniques that can take advantage of having both HMD and HHD closely integrated into one AR system, such as cross-device information sharing, situation adaptive visualization management, and multi-layered visualization.

Mobile Augmented Reality (AR) applications are typically deployed either on head mounted displays (HMD) or handheld displays (HHD). This paper explores novel interaction techniques for a combined HHD-HMD hybrid system that builds on the strengths of each type of device. We use the HMD for viewing AR content and a touch screen HHD for interacting with the content. A prototype system was developed and a user study was conducted comparing four interaction techniques for selection tasks.

In this research, we explore using pointing and drawing in a remote collaboration system. Our application allows a local user with a tablet to communicate with a remote expert on a desktop computer. We compared performance in four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. We found that using drawing annotations would require fewer inputs on an expert side, and would require less cognitive load on the local worker side. In a follow-on study we compared the conditions (2) and (4) using a more complicated task. We found that pointing input requires good verbal communication to be effective and that drawing annotations need to be erased after completing each step of a task.

In this research, we explore how different types of augmented gesture communication cues can be used under different view sharing techniques in a remote collaboration system. In a pilot study, we compared four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. Through this study, we found three results. First, users collaborate more efficiently using annotation cues than pointer cues for communicating object position and orientation information. Second, live video becomes more important when quick feedback is needed. Third, the type of gesture cue has more influence on performance and user preference than the type of view sharing method.

Recent advances in mobile computing and augmented reality (AR) technology have lead to popularization of mobile AR applications. Touch screen input is common in mobile devices, and also widely used in mobile AR applications. However, due to unsteady camera view movement, it can be hard to carry out precise interactions in handheld AR environments, for tasks such as tracing physical objects. In this research, we investigate a Snap-To-Feature interaction method that helps users to perform more accurate touch screen interactions by attracting user input points to image features in the AR scene. A user experiment is performed using the method to trace a physical object, which is typical for modeling real objects within the AR scene. The results shows that the Snap-To-Feature method makes a significant difference in the accuracy of touch screen based AR interaction.

In this paper, we present a prototype Virtual Welding Simulator, which supports interactive training of welding process by using multimodal interface that can deliver realistic experiences. The goal of this research is to overcome difficult problems by using virtual reality technology in training tasks where welding is treated as a principal manufacturing process. The system design and implementation technical issues are described including real-time simulation and visualization of welding bead, providing realistic experience through 3D multimodal interaction, presenting visual training guides for novice workers, and visual and interactive interface for training assessments. According to the results from an initial user study, the prototype VR based simulator system appears to be helpful for training welding tasks, especially in terms of providing visual training guides and instant training assessment.

In this paper, we present a visualization method of weld beads for a welding training under virtual environments. To represent virtual beads, a bead shape is defined according to the datasets which consist of bead width, height, angle, penetration acquired from real welding operations. A curve equation of bead's sectional shape is mathematical modeled, and a height map is generated according to this numerical equation, which is used for generating the bead's mesh data using its height information. Finally, virtual weld beads are visualized in real-time according to the accurately simulated results from user's input motion.

Mobile computing devices are getting popular as a platform for augmented reality (AR) application, and efficient interaction methods for mobile AR environments are considered necessary. Recently, touch interfaces are getting popular and drawing attention as a future standard interface on mobile computing devices. However, accurate touch interactions are not that easy in mobile AR environments where users tend to move and viewpoints easily get shaky. In this paper, the authors suggest a new interaction method for handheld mobile AR environments, named 'Freeze-Set-Go'. The proposed interaction method lets users to 'freeze' the real world view tentatively, and continue to manipulate virtual entities within the AR scene. According to the user experiment, the proposed method turns out to be helping users to interact with mobile AR environments using touch interfaces in a more accurate and comfortable way.

Space planning is one of the popular applications of VR technology including interior design, architecture design, and factory layout. In order to provide easier and efficient methods to accommodate physical objects into virtual space under plan, we suggest applying mixed reality (MR) interface. Our MR system consists of a video see-through display with a touch screen interface, mounted on a mobile platform, and we use screen space 3D manipulations to arrange virtual objects within the MR scene. Investigating the interface with our prototype implementation, we are convinced that our system will help users to design spaces in more easy and effective way.

We present a virtual reality (VR) system that simulates the situation of ship block spray painting. The system was developed under demands from ship building industry for training purpose. We designed an immersive stereo display platform, a realistic spray painting rendering technique and intuitive user interface to match with the real working environment. The system is currently under user-test in the ship building company with receiving positive responses.

Visual information has the greatest effect among five senses of human. Therefore, the satisfaction of visual information for representing virtual environments is necessary for good results in information acquisition, virtual training, virtual prototyping, etc. Recently, although there have been a large variety of display systems, there is no such a fully immersive and clearly projected display with high-resolution. In this paper, we describe a hemispherical display which supports a fully immersive experience and high-resolution images. In our display system for generating high-resolution images, a virtual scene image is divided into several pieces those are rendered by a PC cluster and projected with multiple projectors. In this paper, we also describe the PC cluster and projectors designed for optimized performance and a convenient control.

In this paper, we suggest a new approach for authoring tangible augmented reality applications, called 'immersive authoring.' The approach allows the user to carry out the authoring tasks within the AR application being built, so that the development and testing of the application can be done concurrently throughout the development process. We describe the functionalities and the interaction design for the proposed authoring system that are specifically targeted for intuitive specification of scenes and various object behaviors. Several cases of applications developed using the authoring system are presented. A small pilot user study was conducted to compare the proposed method to a non-immersive approach, and the results have shown that the users generally found it easier and faster to carry out authoring tasks in the immersive environment.

Development of virtual reality systems requires iterations of specification, implementation and evaluation. Since correct evaluations of immersive VR systems require the tedious process of wearing many devices, there exist both temporal and spatial gaps between the implementation and evaluation stage, and this usually causes delay and inefficiency in the development process. In order to overcome this gap, there have been several approaches to constructing or modeling the physical aspects of the virtual world (or objects) within the virtual environment. However, modeling their behaviors is still carried out in conventional (2D) programming environments.This paper proposes an interaction model, and interfaces for specifying (and modifying) object behavior, within the virtual environment, based on an underlying virtual object model. The interaction model follows the concept of programming by demonstration, and based on it, we have built a system called the PiP (Programming virtual object behavior in virtual reality Program) "in" which a user can create, modify, test, and save object behaviors. We illustrate examples of interactive virtual worlds constructed using the PiP, and discuss its merits and shortcomings as a content development platform.