Cross-functional teams struggle when static collaboration tools fail to keep pace with dynamic conversations. Through a formative study with seven professionals, we identified a critical gap: designers and developers speak different vocabularies, causing semantic misalignments. We present Cognitive Bridge, an AI system that monitors multimodal cues (facial expressions, speech, workspace activity) to detect emerging misunderstandings, then generates adaptive boundary objects, visual diagrams, wireframes, and flowcharts that translate between professional perspectives in real-time. Our controlled study with 16 designer-developer dyads found that Cognitive Bridge reduced communication conflicts by 47% and increased implementable solutions by 34% compared to baseline tools. However, analysis revealed a solution-exploration tradeoff: while AI accelerated alignment, it risked premature convergence that constrained creative exploration. We contribute: (1) a novel system for AI-generated boundary objects, and (2) design implications for balancing cognitive scaffolding with creative agency preservation.

This work introduces the Ocular Command Center framework to investigate how eye responses mediate visual effects on physiology and user experience in virtual reality. In a controlled study (N=40), participants experienced variations in luminance, color temperature, peripheral occlusion, and periodic visual suppression while eye activity (pupil size, blinks, fixations, and saccades), cardiovascular responses (heart rate and heart rate variability), and subjective symptoms were measured. Luminance changes affected heart rate through pupillary reflexes. Color temperature affected heart rate variability without pupillary mediation, suggesting appraisal processes, and induced severe nausea. Peripheral occlusion and visual suppression modified oculomotor behavior without substantial cardiovascular effects. These findings demonstrate that visual manipulations could act through distinct reflexive, cognitive, and perceptual pathways, and not all extend equally to systemic physiology. This foundation supports adaptive VR design, regulating comfort, engagement, and physiological state.

The Wizard of Oz (WoZ) method is a common and popular approach for simulating interactive systems in Human-Computer Interaction (HCI). Running such studies is demanding for researchers because the human wizard must manage human–agent interactions in real time while keeping participants safe and the interaction natural. Many WoZ systems struggle to reproduce complex agent behaviours without minimal delays or heavy workload for the moderator. We introduce AI of Oz, a framework that uses large language models to support researchers by monitoring ongoing interactions, detecting sensitive moments, and suggesting contextually appropriate responses. In a study with 20 HCI-related researchers, the system improved participants’ ability to manage interactions and maintain control compared to a version without AI support. We outline implications for WoZ research and note current limitations and future directions.

The emergence of generative artificial intelligence (GenAI), combined with immersive virtual reality (VR), enables the rapid creation of personalised virtual content from simple text prompts, holding potential for emotional support. However, most current VR systems rely on pre-authored content and limit user agency in designing emotionally meaningful experiences. We introduce OasisMind, an AI-assisted VR system that empowers users to co-create 360° environments, corresponding ambient soundscapes, and context-aware digital companions through natural language prompts. In a user study (N=24), we observed how participants constructed virtual worlds for emotionally meaningful use cases and compared their creations to validated, pre-defined VR scenes recommended by previous research. Our results indicate a subjective preference for self-created environments, while no significant differences were observed in perceived satisfaction or presence between conditions. These findings suggest that user agency contributes to the emotional resonance of virtual experiences and inform the design of future personalised companion systems.

Integrating non-verbal cues into Mixed Reality Agents (MiRAs) enhances their ability to engage users and foster socially rich interactions. This paper investigates the role of locomotion and body posture in shaping user engagement, social presence, and interaction quality through two user studies involving a turn-based Gobang game. From these studies we found that in a competitive context, MiRAs’ locomotion and posture enhanced social presence and engagement, but while in a cooperative context, these behaviors fostered rapport but not trust. By integrating subjective, behavioral, and physiological measures, including EEG, this study provides a holistic understanding of MiRAs’ impact. The findings offer actionable design implications for creating engaging and socially effective virtual agents, advancing the field of human-agent interaction in Mixed Reality. Future research directions include exploring long-term effects, using diverse application domains, and supporting multimodal interactions.

Conversational agents (CAs) are transforming human-computer interaction, evolving from text-based chatbots to digital humans (DHs) capable of rich emotional expression. This study explores integrating neural and physiological signals into the perception module of CAs to enable real-time emotion detection and empathetic responses. We conducted a user study in which participants engaged with a DH about emotional topics. The DH mirrored participants’ emotions in real-time using neural and physiological cues. Results showed that users experienced stronger emotions and greater engagement during interactions with the Empathetic DH, highlighting the benefits of these signals for enhancing empathy. However, challenges remain, including recognition accuracy, emotional transition timing, individual differences, and limited voice modulation. Addressing these issues is key to advancing empathetic digital agents. This research demonstrates the promise of real-time physiological and neural emotion recognition for building emotionally intelligent CAs that foster deeper, more meaningful human-agent interactions.

In Mixed Reality (MR) remote collaboration for industrial tasks, it’s essential to offer clear visual engineering information through a user-friendly human-computer interaction interface for collaborators. Many research prototypes have demonstrated transmitting hand-drawn sketches based on gesture-based interaction, but there has been little research on remote collaboration on procedural tasks exploring the influence of sharing hand-drawn sketches and gesture on the local spatial AR (SAR) site and supporting passive haptic feedback on the remote VR site. Therefore, in this paper, we propose a Mixed Reality remote collaboration system based on sketch cues that can offer these capabilities. The system takes advantage of VR/AR technologies to change the method engineers interact with the visual interface and convey engineering informatics. The research findings showed there were significant differences in performance efficiency and perception experience, but not for operational errors.

Video conferencing is essential for remote collaboration, but it often leads to fatigue, reduced social presence and ineffective communication. Traditional human facilitators can address these challenges but cannot scale to meet the demands of countless daily virtual meetings across organisations. To address these challenges, we introduce Cognitive Load and Affect Aware Agent (CLARA), an AI-mediated facilitator that enhances group decision-making by dynamically managing cognitive load and affective engagement. CLARA employs real-time multimodal assessment of group states, providing adaptive cognitive prompts to optimise task focus and affective cues to foster positive dynamics. In a controlled study (N = 48), we compared Baseline, Cognitive Feedback (CF), Affective Feedback (AF) and Combined Feedback (CAF) conditions. Results show CAF significantly improved task performance, reduced mental demand and enhanced social presence, outperforming all other conditions. Participants rated CAF as having the highest level of facilitator expertise and preference. These findings highlight the benefits of integrated AI-driven facilitation, offering design insights for human-centred, effective virtual collaboration tools that balance task efficiency with positive socio-emotional engagement.

Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1stJoint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

Mixed Reality Agents (MiRAs) have been extensively studied to enhance virtual-physical interactions, using their ability to exist in both virtual and physical environments. However, little research has focused on enhancing perceived empathy in MiRAs, despite its potential for agent-assisted therapy, education, and training. To fill this gap, we investigate the impact of an Empathic Mixed Reality agent (EMiRA) that adapts to users' physiological states and physical events in a shooting game. We found that this adaptation enhanced users' social perceptions of the agent, including social presence, social connectedness, and perceived empathy. Physiological adaptation increased paternalism and reduced user dominance, while physical adaptation had no such effect. We discuss these findings and provide design implications for future EMiRAs.

We introduce bichronous collaboration in an eXtended Reality (XR) remote collaboration workflow that seamlessly links asynchronous review with synchronous cooperation on physical tasks. The aim is to reduce the misunderstandings and errors that often arise during remote work. The system records a novice's actions and presents them with three spatiotemporal cues-2D, 3D Basic, and 3D Action. An expert can review and provide feedback asynchronously, and also guide the novice synchronously to fix the error. A user study showed that the 3D Action cue allowed experts to identify faults more quickly, enabled novices to resolve problems in less time with higher success rates than a conventional 2D timeline, without increasing cognitive workload or cybersickness. These findings highlight the value of coupling asynchronous insight with synchronous teamwork and position bichronous collaboration as a practical solution for remote troubleshooting, training, and inspection in XR environments.

Augmented Reality (AR) has demonstrated significant potential in task guidance by offering hands-free, context-sensitive instructions. However, effective AR-based task guidance requires maintaining user focus, as lapses in attention can lead to inefficiencies and errors. This paper describes a novel approach for integrating real-time gaze and location tracking to dynamically adapt AR task guidance, adjusting the playback speed accordingly. A user study with 20 participants assessed the effectiveness of gaze-based and locationbased focus detection across four conditions: baseline, gaze-only, location-only, and combined interaction. The results indicate that both gaze-based and location-based guidance significantly reduce error rates and improve task efficiency, however, the simultaneous use of both methods leads to cognitive overload, increased errors, and interaction complexity. Participants found that the adaptive playback speed adjustments helped manage attention. These findings highlight the importance of balancing adaptive cues in AR task guidance to optimize user performance without overwhelming cognitive resources.

We introduce HeadZoom, a hands-free interaction technique for navigating two-dimensional visual content using head movements. HeadZoom enables fluid zooming and panning using only real-time head tracking. It supports natural control in applications such as map exploration, radiograph inspection, and image browsing, where physical interaction is limited. We evaluated HeadZoom in a withinsubjects study comparing three interaction techniques-Static, Tilt Zoom, and Parallel Zoom-across spatial, error, and subjective metrics. Parallel Zoom significantly reduced total head movement compared to Static and Tilt modes. Users reported significantly lower perceived exertion for Parallel Zoom, confirming its suitability for prolonged or precision-based tasks. By minimizing movement demands while maintaining task effectiveness, HeadZoom advances the design of head-based 2D interaction in VR and creates new opportunities for accessible hands-free systems for image exploration.

Non-verbal cues like locomotion and posture influence users’ perceptions of Mixed Reality Agents (MiRAs). While Electroencephalography (EEG) captures cognitive responses, the influence of MiRAs’ locomotion and postures on brain activity remains underexplored. Additionally, few studies integrate subjective and behavioral measures with EEG to evaluate these cues’ impact on social perception. To address this, we conducted a within-subject study where participants played Gobang against three virtual agents in mixed reality: 1) a speech-only agent (S), 2) an embodied agent with speech and locomotion (S + L), and 3) an embodied agent with speech, locomotion, and posture (S + L + P). Results showed the S + L + P agent had higher engagement measured by the questionnaire but a lower EEG-based engagement index at AF3 than the S + L agent. Besides, the S + L + P was also rated higher in social presence, engagement, and emotional arousal than the S condition; No behavioral differences were observed. We discuss how MiRAs’ locomotion and posture affect users’ social perception and provide design implications for future human-agent interactions.

Prior research has largely focused on AR or VR alone, but recent CR advances show the benefit of fluidly combining them. This article presents a novel cross-reality (CR) task guidance system designed to support asynchronous task guidance on spatial tasks. The system enables (1) recording and playback of user actions within large-scale workspaces and (2) seamless transitions between augmented reality (AR), world-in-miniature (WIM), and virtual reality (VR) interfaces. A user study demonstrated that the CR interface significantly improved task efficiency, reduced required physical activity, enhanced spatial memory recall, and lowered perceived task load. These findings highlight the system’s ability to bridge temporal and spatial gaps in asynchronous task guidance and suggest the broader potential of CR technologies for collaborative workflows. Limitations and design implications are also discussed to inform future research on CR interface development.

Manual precise manipulation of objects is a critical skill in daily life, and Augmented Reality (AR) is increasingly used to support such tasks. In this study, we propose a system utilizing pseudo-haptic feedback to support precise manipulation for six degrees of freedom (6DOF). Two types of AR instruction interfaces were developed: Visual Deviation Instruction Interface (VDI) and Pseudo-Haptic Instruction Interface (PHI). A user study with 18 participants compared the two instruction interfaces in terms of performance and user experience. The objective measures of performance (task completion time, deviation), and the subjective measures (system usability scale, NASA Task Load Index) were collected. Results show that both instruction interfaces effectively support manual precise manipulation, achieving position deviations under 2 mm and orientation deviations under 1°. PHI outperformed VDI in speed, mental effort, physical demand, performance, perceived workload, custom user experience elements and reduction deviations of manual precise manipulation. Finally, we discuss research limitations and future directions.

Autistic children may often experience challenges in mastering daily living skills crucial for their independence and well-being. This study introduces “Tidd,” an augmented tabletop interactive system designed to assist autistic children in practising daily living skills in an engaging and physically interactive environment. We conducted a user study in a medical rehabilitation centre and an integrated kindergarten. Seventeen autistic children aged three to five years used Tidd in training sessions covering two vital skills: bed-making and dressing. Progress was evaluated through task completion and progress tracking, observational data for children, and therapist qualitative feedback. Therapists reported that Tidd was beneficial in maintaining the children’s attention and enhancing their motivation. Observational data further suggested increased engagement and decreased frustration during tasks. This study with Tidd highlights the potential of tabletop interaction to support therapists in training autistic children to learn daily living skills. • Tidd: AR tabletop device aiding autistic children in bed-making and dressing. • Improved task accuracy and engagement in study with 17 autistic children. • Therapists validated Tidd’s practicality and appeal in autism interventions.

Understanding the relationship between cognitive load and affective state in remote work is vital for designing intuitive collaboration. We present CoAffinity, a multimodal dataset encompassing eight structured remote-work tasks, during which 39 participants provided self-reported measures (arousal, valence, positive/negative affect, and cognitive-load) while being recorded via audio, video, and physiological signals (PPG and GSR). Spanning over 38 hours of annotated data, our approach involved precise timestamp alignment, short and long-session labelling, and subsequent machine-learning and deep-learning benchmarks. Key findings show that integrating multiple modalities, especially physiological data, significantly improves the detection of cognitive load and emotion, while group synchrony metrics highlight how physiological coherence shifts under varied task demands. By capturing complex cognitive-emotional dynamics in realistic remote settings, CoAffinity aims to advance affective computing, inform human-computer interaction research, and foster more empathetic remote collaboration tools

This paper presents FEAD, a fNIRS-EEG Affective Database that can be used for training emotion recognition models. The electrical activity and brain hemodynamic responses of 37 participants were recorded, as well as the categorical and dimensional emotion ratings they gave to 24 affective audio-visual stimuli. The relationship between the neurophysiological signals with the subjective ratings was investigated, with a significant correlation found in the prefrontal cortex region. A binary classification of affective states was performed using a subject-dependent approach, taking into account the fusion of both modalities (functional Near-Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG)) and each single modality separately. In addition, we explored the temporal dynamics of the recorded data in shorter trials and found that the fusion of features from both modalities yielded significantly better results than using a single modality. This database will be made publicly available with the aim to encourage researchers to develop more advanced algorithms for affective computing and emotion recognition.

Virtual content in augmented reality (AR) applications can be tailored to a designer's specifications. However, real-world environments are challenging to control precisely or replicate fully. Consequently, prototyping AR applications for specific environments is often difficult. One potential solution is employing mixed reality (MR) to simulate an AR system, enabling controlled experiments. Nevertheless, the effectiveness of using MR to simulate AR office work remains underexplored. In this paper, we report the results of a user study (N = 40) that investigated the impact of an MR simulation of an AR office on participants' task performance and cognitive workload (CWL). Participants completed several office tasks in both an AR scene featuring a virtual monitor and an MR-simulated AR scene. During these tasks, CWL was measured using electroencephalography (EEG) and a subjective questionnaire. The results show that the performance of the pass-through window is a major constraint on the effectiveness of the MR simulation office. Finally, we discuss the study's limitations and directions for future research.

In human-agent interaction, establishing trust and a social bond with the agent is crucial to improving communication quality and performance in collaborative tasks. This paper investigates how a Mixed Reality Agent’s (MiRA) ability to acknowledge a user’s physiological state affects perceptions such as empathy, social connectedness, presence, and trust. In a within-subject study with 24 subjects, we varied the companion agent’s awareness during a mixed-reality first-person shooting game. Three agents provided feedback based on the users’ physiological states: (1) No Awareness Agent (NAA), which did not acknowledge the user’s physiological state; (2) Random Awareness Agent (RAA), offering feedback with varying accuracy; and (3) Accurate Awareness Agent (AAA), which provided consistently accurate feedback. Subjects reported higher scores on perceived empathy, social connectedness, presence, and trust with AAA compared to RAA and NAA. Interestingly, despite exceeding NAA in perception scores, RAA was the least favored as a companion. The findings and implications for the design of MiRA interfaces are discussed, along with the limitations of the study and directions for future work.

Representing emotion in collaborative Virtual Reality (VR) environments is an emerging topic, as VR can enable humans to express augmented emotions beyond their normal abilities. This research explores how emotion can be represented in collaborative VR beyond facial expressions. We developed a virtual system that communicates emotion using three sensory modalities (textual, auditory and visual) through two spatiotemporal representations (human-form avatar and superpower). We show real-time emotion through a natural avatar and objectify emotion states into superpower phenomena in in-situ environments for time periods. We incorporated subjective, physiological and behavioural measures to evaluate emotion in an asynchronous VR collaboration scenario. The results suggested that showing emotions through the avatar and superpower augmentations (audio-visual) provided the best immersive VR experience, where users were more aroused, and the positive emotion felt more dominating. We also found that understanding emotion requires easy and relatable visuals that people commonly acknowledge, whereas arousing emotion requires a change of environmental contexts to indicate different states. We provide design insights for using multisensory modalities in empathic VR systems to address the lack of a standardised representation of emotion in collaborative VR.

In this research, we investigate the effectiveness of asymmetric interactions (HandStylus, HandController, and TwoHands) in Augmented Reality (AR), Virtual Reality (VR), and Extended Reality (XR) for 3D digital drawing overlaying on physical and virtual objects. We evaluate the input accuracy and fatigue of these object-based 3D drawing experiences using quantitative measurements and further explore the correlation between these outcomes with subjective questionnaires. We found significant independence between environments and interface types, which considerably influence the performance and usability of 3D immersive drawing. We noted discrepancies between users’ subjective experiences and objective performance. Specifically, although AR drawing on physical objects provides superior accuracy and minimal muscle fatigue due to tangible feedback, and the TwoHands interaction offers the highest precision, the subjective results show the reverse outcome. Based on these findings, we propose design recommendations and discuss directions for future research in immersive drawing environments.

This paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability, and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.

The 2024 IEEE VGTC Virtual Reality Service Award goes to Mark Billinghurst of the University of South Australia and the University of Auckland, in recognition of his many years of service contributions to the AR/VR academic community. Prof. Billinghurst has most recently served as the chair of the IEEE ISMAR steering committee and currently serves on the IEEE VR steering committee. He has served in leadership roles for the IEEE VR and IEEE ISMAR conferences, including as general chair of ISMAR twice. Over his career, Prof. Billinghurst has served in over 100 service roles for VGTC conferences. The IEEE VGTC is pleased to award Mark Billinghurst the 2024 Virtual Reality Service Award.

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees—with special recognition for the multiple years of hard work of the General Chairs, Carolina Cruz-Neira, Greg Welch, and Xubo Yang. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference’s status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

In collaborative settings where multiple individuals are tasked with completing a shared goal, understanding one’s partner’s emotional state could be crucial for achieving a successful outcome. This is particularly relevant in remote collaboration contexts, where physical distance can impede understanding, empathy, and mutual comprehension between partners. In this paper, we demonstrate representing emotional patterns from physiological data in a shared Virtual Reality (VR) environment, and explore how it impacted communication styles. A user study investigated the potential effects of this emotional representation in fostering empathetic communication during remote collaboration. The study’s findings revealed that although there was minimal variance in the workload associated with observing physiological cues, participants generally preferred monitoring their partner’s attentional state. However, with the assembly task chosen, most participants only directed a minimal proportion of their attention toward the physiological cues displayed by their partner, and were frequently uncertain of how to interpret and use the information obtained. We also discuss limitations of the research and opportunities for future work.

Augmented reality (AR) can bring a new interactive experience to the collaboration between users. When users are in the same place, there are two modes of joint operation for the same object: parallel-work (PW) and cross-work (CW). PW means two users perform their tasks, while CW means assisting each other. To investigate the difference that collaboration using PW and CW in an AR environment brings to users, we developed a two-person local collaboration system, LoCol. We designed and conducted user experiments by selecting the tasks of adjusting the virtual model of the assembly and adding missing boundaries in the model. The results showed that CW led to a higher sense of social coexistence while reducing workload. In terms of task completion time and accuracy, CW and PW each had advantages. We found that users generally want to reduce unnecessary repetitive operations and frequent movement by working with others. This is likely an important criterion for determining who is better suited for a particular job in either approach.

The shortcomings of established input methods for Augmented Reality (AR) head-mounted displays (HMDs) motivate us to investigate the use of AR HMDs with smartphones and smartwatches to improve AR interaction in physical tasks. However, it is unclear whether the cross-device interaction interfaces are efficient for AR systems on physical tasks because the physical tasks can break the interaction flow. In this work, we conducted a user study to explore this. The user study consists of three subtasks, respectively requiring one of the three representative user interface (UI) controls (buttons, sliders, or text input). We implemented them with mid-air gestures, a smartphone, and a smartwatch. We compared the three interfaces and found that the smartphone and the smartwatch interaction interfaces have greater usability, provide a better user experience, and have lower workloads than the mid-air gesture interface. However, there is no significant difference in terms of efficiency for simple interactions. We discuss the limitations of this research and directions for future work.

Interaction with virtual objects is one of the essential features of Augmented Reality (AR) systems. One of its main issues is how to provide precise manipulation of distant virtual objects in AR. In this work, we explore bare-hand manipulation of distant objects in AR with DOF (degree-of-freedom) separation, motion scaling, and near-field metaphors. We developed two manipulation techniques: the distant widget-based metaphor (DWBM), and the near-field widget-based metaphor (NFWBM). We conducted a user study with 20 participants to compare the two techniques in terms of performance and user experience. We found that NFWBM has faster speed, lower mental effort, better ease-of-use, and a friendlier user experience. However, there is no significant difference in terms of precision; both techniques could manipulate distant objects precisely, with an average position error of less than 0.7 cm and an average orientational error of less than 1°. We also discussed the limitations of this research and directions for future work.

This research investigates fast and precise Virtual Reality (VR) sketching methods with different tool-based asymmetric interfaces. In traditional real-world drawing, artists commonly employ an asymmetric interaction system where each hand holds different tools, facilitating diverse and nuanced artistic expressions. However, in virtual reality (VR), users are typically limited to using identical tools in both hands for drawing. To bridge this gap, we aim to introduce specifically designed tools in VR that replicate the varied tool configurations found in the real world. Hence, we developed a VR sketching system supporting three hybrid input techniques using a standard VR controller, a VR stylus, or a data glove. We conducted a formal user study consisting of an internal comparative experiment with four conditions and three tasks to compare three asymmetric input methods with each other and with a traditional symmetric controller-based solution based on questionnaires and performance evaluations. The results showed that in contrast to symmetric dual VR controller interfaces, the asymmetric input with gestures significantly reduced task completion times while maintaining good usability and input accuracy with a low task workload. This shows the value of asymmetric input methods for VR sketching. We also found that the overall user experience could be further improved by optimizing the tracking stability of the data glove and the VR stylus.

When using Virtual Reality (VR) and Augmented Reality (AR) to support remote collaboration, effective communication between a remote expert in VR and a local worker in AR is important for guiding and following task instructions. This is especially crucial for assembly tasks, which require precise identification of parts and clear directions for their combination. Despite the increasing interest in efficient VR-AR communication methods, previous studies have been limited to complex hardware setups and simplified assembly tasks. In this research, we introduce a communication approach for remote collaboration in complex assembly tasks, utilizing simplified hardware configurations. We conducted a user study (n=30) and compared three interaction interfaces (hand gestures, 3D drawing, and virtual replicas) in task completion time, subjective questionnaires, and preference rank. The results showed that the use of virtual replicas not only enhances task efficiency but also receives strong preference by users. These findings indicate that virtual replicas can provide intuitive instructions to local workers, resulting in a clearer understanding of the expert’s guidance.

We introduce RadarHand, a wrist-worn wearable with millimetre wave radar that detects on-skin touch-based proprioceptive hand gestures. Radars are robust, private, small, penetrate materials, and require low computation costs. We first evaluated the proprioceptive and tactile perception nature of the back of the hand and found that tapping on the thumb is the least proprioceptive error of all the finger joints, followed by the index finger, middle finger, ring finger, and pinky finger in the eyes-free and high cognitive load situation. Next, we trained deep-learning models for gesture classification. We introduce two types of gestures based on the locations of the back of the hand: generic gestures and discrete gestures. Discrete gestures are gestures that start at specific locations and end at specific locations at the back of the hand, in contrast to generic gestures, which can start anywhere and end anywhere on the back of the hand. Out of 27 gesture group possibilities, we achieved 92% accuracy for a set of seven gestures and 93% accuracy for the set of eight discrete gestures. Finally, we evaluated RadarHand’s performance in real-time under two interaction modes: Active interaction and Reactive interaction. Active interaction is where the user initiates input to achieve the desired output, and reactive interaction is where the device initiates interaction and requires the user to react. We obtained an accuracy of 87% and 74% for active generic and discrete gestures, respectively, as well as 91% and 81.7% for reactive generic and discrete gestures, respectively. We discuss the implications of RadarHand for gesture recognition and directions for future works.

There is little research on how Virtual Reality (VR) applications can identify and respond meaningfully to users' emotional changes. In this paper, we investigate the impact of Context-Aware Empathic VR (CAEVR) on the emotional and cognitive aspects of user experience in VR. We developed a real-time emotion prediction model using electroencephalography (EEG), electrodermal activity (EDA), and heart rate variability (HRV) and used this in personalized and generalized models for emotion recognition. We then explored the application of this model in a context-aware empathic (CAE) virtual agent and an emotion-adaptive (EA) VR environment. We found a significant increase in positive emotions, cognitive load, and empathy toward the CAE agent, suggesting the potential of CAEVR environments to refine user-agent interactions. We identify lessons learned from this study and directions for future work.

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches would certainly benefit from an accurately labeled, real-world, diverse dataset that enables the development of generalizable ML models. We introduce 'VR.net', a dataset comprising 165-hour gameplay videos from 100 real-world games spanning ten diverse genres, evaluated by 500 participants. VR.net accurately assigns 24 motion sickness-related labels for each video frame, such as camera/object movement, depth of field, and motion flow. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we implement a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.We also provide access to our data collection tool, enabling researchers to contribute to the expansion of VR.net.

With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner’s mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and co-presence to an audio-only baseline. In addition, we tracked the musicians’ physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.

This paper presents a novel solution for estimating simulator sickness in HMDs using machine learning and 3D motion data, informed by user-labeled simulator sickness data and user analysis. We conducted a novel VR user study, which decomposed motion data and used an instant dial-based sickness scoring mechanism. We were able to emulate typical VR usage and collect user simulator sickness scores. Our user analysis shows that translation and rotation differently impact user simulator sickness in HMDs. In addition, users’ demographic information and self-assessed simulator sickness susceptibility data are collected and show some indication of potential simulator sickness. Guided by the findings from the user study, we developed a novel deep learning-based solution to better estimate simulator sickness with decomposed 3D motion features and user profile information. The model was trained and tested using the 3D motion dataset with user-labeled simulator sickness and profiles collected from the user study. The results show higher estimation accuracy when using the 3D motion data compared with methods based on optical flow extracted from the recorded video, as well as improved accuracy when decomposing the motion data and incorporating user profile information.

Quick visual design• Capture key interactions • Focus on user experience • Communicate design ideas • "Learn by doing/experiencing"

The IEEE VR Steering Committee congratulates and offers gratitude to the enormous efforts of the IEEE VR Conference Organizing Committees — with special recognition for the multiple years of hard work of the General Chairs, Xubo Yang, Kun Zhou, Tobias Langlotz, and Stephan Lukosch. To realize the conference requires a sizeable, motivated team of volunteers to work with the IEEE conference management team. Maintaining the conference's status as the premier international virtual reality conference is a testament to the commitment of all the organizers and participants that make up the IEEE Virtual Reality community.

Many Virtual Reality (VR) experiences, such as learning tools, would benefit from utilising mental states such as cognitive load. Increases in cognitive load (CL) are often reflected in the alteration of physiological responses, such as pupil dilation (PD), electrodermal cctivity (EDA), heart rate (HR), and electroencephalography (EEG). However, the relationship between these physiological responses and cognitive load are usually measured while participants sit in front of a computer screen, whereas VR environments often require a high degree of physical movement. This physical activity can affect the measured signals, making it unclear how suitable these measures are for use in interactive Virtual Reality (VR).

Brain–computer interfaces (BCIs) are an emerging technology with numerous applications. Electroencephalogram (EEG) motor imagery (MI) is among the most common BCI paradigms and has been used extensively in healthcare applications such as post-stroke rehabilitation. Using a Virtual Reality (VR) game, Push Me, we conducted a pilot study to compare MI accuracy with Gel or active-dry EEG electrodes. The motivation was to (1) investigate the MI paradigm in a VR environment and (2) compare MI accuracy using active dry and gel electrodes with different Machine Learning (ML) classifications (SVM, KNN and RF). The results indicate that while gel-based electrodes, in combination with SVM, achieved the highest accuracy, dry electrode EEG caps achieved similar outcomes, especially with SVM and KNN models.

This paper explores the development and testing of fabric electrodes to collect a range of physiological measures. The aim is to integrate these sensors into a Virtual Reality (VR) headset to collect physiological and muscular motion data that will help detect emotion, cognitive load and facial expressions. As part of an on-going project, we have already developed prototypes of the EMG and GSR sensors. A head phantom has been developed for the purpose of testing and validating electrode performance.

Augmented Reality (AR) based manual assembly nowadays enables to guide the process of physical tasks, providing intuitive instructions and detailed information in real-time. However, very limited studies have explored AR manual adjustment tasks with precision requirements. In this paper, we develop an AR-assisted guidance system for manual adjustments with relatively high-precision requirements. We first assessed the accuracy of the special-set OptiTrack system to determine the threshold of precision requirements for our user study. We further evaluated the performance of Number-based and Bar-based precision feedback by comparing orienting assembly errors and task completion time, as well as the usability in the user study. We found that the assembly errors of orientation in the Number-based and Bar-based interfaces were significantly lower than the baseline condition, while there was no significant difference between the Number-based and Bar-based interfaces. Furthermore, the Number-based showed faster task completion time, lower workload, and higher usability than the Bar-based condition.

Physics-driven predictions of soft tissue mechanics are vital for various medical interventions. Insights on the mechanical properties of soft tissues are essential for obtaining personalised predictions from these models. This study aims to provide a workflow to identify the material parameters of soft homogeneous materials under gravity loading using 3D surface geometrical measurements acquired from a wearable augmented reality (AR) headset’s depth camera. Preliminary results show that the parameter estimation procedure can successfully recover the ground truth material parameter C1 of a cantilever beam using synthetic surface data. This workflow could be used for real-time navigational guidance during soft tissue treatment procedures.

During component assembly, some operations must be completed by two or more workers due to the size or assembly mode. For instance, in manual riveting, two workers are positioned on either side of a steel plate, which blocks the view. Traditional collaborative approaches limit assembly efficiency and is difficult to ensure accurate and rapid interaction between workers. In this study, we developed an AR-Assisted Cooperative Assembly System (ARCoA) to address the issue. ARCoA allows users to view their partner’s key information that is occluded, including tools, gestures, orientations, and shared markers. Besides, we presented a user experiment that compared this method with the traditional approach. The results indicated that the new system could significantly improve assembly efficiency, system availability, and sense of social presence. Moreover, most users we surveyed preferred ARCoA. In the future, we will incorporate more functions and improve the accuracy of the system to solve complex multi-person collaborative problems.

Consistent visual and haptic feedback is an important way to improve the user experience when interacting with virtual objects. However, the perception provided in Augmented Reality (AR) mainly comes from visual cues and amorphous tactile feedback. This work explores how to simulate positional vibrotactile feedback (PVF) with multiple vibration motors when colliding with virtual objects in AR. By attaching spatially distributed vibration motors on a physical haptic proxy, users can obtain an augmented collision experience with positional vibration sensations from the contact point with virtual objects. We first developed a prototype system and conducted a user study to optimize the design parameters. Then we investigated the effect of PVF on user performance and experience in a virtual and real object alignment task in the AR environment. We found that this approach could significantly reduce the alignment offset between virtual and physical objects with tolerable task completion time increments. With the PVF cue, participants obtained a more comprehensive perception of the offset direction, more useful information, and a more authentic AR experience.

In this paper, we explore how virtual replicas can enhance Mixed Reality (MR) remote collaboration with a 3D reconstruction of the task space. People in different locations may need to work together remotely on complicated tasks. For example, a local user could follow a remote expert's instructions to complete a physical task. However, it could be challenging for the local user to fully understand the remote expert's intentions without effective spatial referencing and action demonstration. In this research, we investigate how virtual replicas can work as a spatial communication cue to improve MR remote collaboration. This approach segments the foreground manipulable objects in the local environment and creates corresponding virtual replicas of physical task objects. The remote user can then manipulate these virtual replicas to explain the task and guide their partner. This enables the local user to rapidly and accurately understand the remote expert's intentions and instructions. Our user study with an object assembly task found that using virtual replica manipulation was more efficient than using 3D annotation drawing in an MR remote collaboration scenario. We report and discuss the findings and limitations of our system and study, and present directions for future research.

Wearable Augmented Reality (AR) has attracted considerable attention in recent years, as evidenced by the growing number of research publications and industry investments. With swift advancements and a multitude of interdisciplinary research areas within wearable AR, a comprehensive review is crucial for integrating the current state of the field. In this paper, we present a review of 389 research papers on wearable AR, published between 2018 and 2022 in three major venues: ISMAR, TVCG, and CHI. Drawing inspiration from previous works by Zhou et al. and Kim et al., which summarized AR research at ISMAR over the past two decades (1998-2017), we categorize the papers into different topics and identify prevailing trends. One notable finding is that wearable AR research is increasingly geared towards enabling broader consumer adoption. From our analysis, we highlight key observations related to potential future research areas essential for capitalizing on this trend and achieving widespread adoption. These include addressing challenges in Display, Tracking, Interaction, and Applications, and exploring emerging frontiers in Ethics, Accessibility, Avatar and Embodiment, and Intelligent Virtual Agents.

Digital humans are a highly realistic form of conversational computer agent. Eye gaze is a salient social cue that digital humans could use to facilitate rapport-building during conversations. However, eye gaze tendencies vary by gender and incorrect gaze patterns can have negative social implications. Analysis of observational data during human conversations can help inform the development of eye gaze models for digital humans. This study aimed to identify the eye gaze patterns of women dyads during a rapport-building conversation, and to evaluate the effect of different gaze patterns on rapport, trust, and psychological outcomes. 36 adult women (18 dyads) completed the Relationship Closeness Induction Task while wearing eye tracking glasses. Subjective rapport, trust, and psychological measures were collected. Gaze patterns of women were found to change as the conversation content became more intimate; specifically, gaze aversions for thinking (p=.042), turn-taking (p=.025), and intimacy modulation increased in duration (p=.012). Furthermore, gaze patterns were associated with perceptions of the conversation partner. Displaying fewer cognitive gaze aversions was associated with greater closeness (p=.029) and trust perceptions (p=.035). Longer periods of direct gaze while speaking was associated with greater rapport (p=.040). Results will inform the development of a humanlike gaze model for female digital humans during intimate conversations and may be applicable to social robots.

In this paper, we share real-time collaborative gaze behaviours, hand pointing, gesturing, and heart rate visualisations between remote collaborators using a live 360 ° panoramic-video based Mixed Reality (MR) system. We first ran a pilot study to explore visual designs to combine communication cues with biofeedback (heart rate), aiming to understand user perceptions of empathic collaboration. We then conducted a formal study to investigate the effect of modality (Gaze+Hand, Hand-only) and interface (Near-Gaze, Embodied). The results show that the Gaze+Hand modality in a Near-Gaze interface is significantly better at reducing task load, improving co-presence, enhancing understanding and tightening collaborative behaviours compared to the conventional Embodied hand-only experience. Ranked as the most preferred condition, the Gaze+Hand in Near-Gaze condition is perceived to reduce the need for dividing attention to the collaborator’s physical location, although it feels slightly less natural compared to the embodied visualisations. In addition, the Gaze+Hand conditions also led to more joint attention and less hand pointing to align mutual understanding. Lastly, we provide a design guideline to summarize what we have learned from the studies on the representation between modality, interface, and biofeedback.

This paper presents XRtic, a prototyping toolkit enabling real-world cloth deformations to be used in novel ways in eXtended Reality (XR) applications. XRtic was developed based on the insights gathered from semi-structured interviews with XR developers. It consists of custom-made actuators that can be attached to regular clothing, a controller bus system, and a controller interface. Using our toolkit, users can design and integrate different cloth deformation types synchronised with virtual content in a plug-and-play manner. Along with a technical analysis of the actuation behaviour of the XRtic actuators, we present the findings gathered from a user study with eight XR developers, focusing on the usability of the system and creative support. Overall, participants found it an easy-to-use toolkit that supports iterative and rapid prototyping, and enables cloth to be deformed in unique ways in synchronisation with XR applications. Based on the findings, we also report limitations and future work relating to our system.

Emotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically recognizing emotions in conversation could be applied in many application domains like health-care, education, social interactions, entertainment, and more. Facial expressions, speech, and body gestures are primary cues that have been widely used for recognizing emotions in conversation. However, these cues can be ineffective as they cannot reveal underlying emotions when people involuntarily or deliberately conceal their emotions. Researchers have shown that analyzing brain activity and physiological signals can lead to more reliable emotion recognition since they generally cannot be controlled. However, these body responses in emotional situations have been rarely explored in interactive tasks like conversations. This paper explores and discusses the performance and challenges of using brain activity and other physiological signals in recognizing emotions in a face-to-face conversation. We present an experimental setup for stimulating spontaneous emotions using a face-to-face conversation and creating a dataset of the brain and physiological activity. We then describe our analysis strategies for recognizing emotions using Electroencephalography (EEG), Photoplethysmography (PPG), and Galvanic Skin Response (GSR) signals in subject-dependent and subject-independent approaches. Finally, we describe new directions for future research in conversational emotion recognition and the limitations and challenges of our approach.

Despite the increasing popularity of VR games, one factor hindering the industry’s rapid growth is motion sickness experienced by the users. Symptoms such as fatigue and nausea severely hamper the user experience. Machine Learning methods could be used to automatically detect motion sickness in VR experiences, but generating the extensive labeled dataset needed is a challenging task. It needs either very time consuming manual labeling by human experts or modification of proprietary VR application source codes for label capturing. To overcome these challenges, we developed a novel data collection tool, VRhook, which can collect data from any VR game without needing access to its source code. This is achieved by dynamic hooking, where we can inject custom code into a game’s run-time memory to record each video frame and its associated transformation matrices. Using this, we can automatically extract various useful labels such as rotation, speed, and acceleration. In addition, VRhook can blend a customized screen overlay on top of game contents to collect self-reported comfort scores. In this paper, we describe the technical development of VRhook, demonstrate its utility with an example, and describe directions for future research.

In this paper, we present a 360° panoramic Mixed Reality (MR) sys-tem that visualises shared gaze cues using contextual speech input to improve task coordination. We conducted two studies to evaluate the design of the MR gaze-speech interface exploring the combinations of visualisation style and context control level. Findings from the first study suggest that an explicit visual form that directly connects the collaborators’ shared gaze to the contextual conversation is preferred. The second study indicates that the gaze-speech modality shortens the coordination time to attend to the shared interest, making the communication more natural and the collaboration more effective. Qualitative feedback also suggest that having a constant joint gaze indicator provides a consistent bi-directional view while establishing a sense of co-presence during task collaboration. We discuss the implications for the design of collaborative MR systems and directions for future research.

This work investigates the use of Virtual Reality (VR) to present forensic evidence to the jury in a courtroom trial. The findings of a between-participant user study on comprehension of an expert statement are presented, examining the benefits and issues of using VR compared to traditional courtroom presentation (being still images). Participants listened to a forensic scientist explain bloodstain spatter patterns while viewing a mock crime scene in either VR or as still images in video format. Under these conditions, we compared understanding of the expert domain, mental effort and content recall. We found that VR significantly improves the understanding of spatial information and knowledge acquisition. We also identify different patterns of user behaviour depending on the display method. We conclude with suggestions on how to best adapt evidence presentation to VR.

We present a Virtual Reality (VR) game, PlayMeBack, to investigate cognitive load measurement in interactive VR environments using pupil dilation, Galvanic Skin Response (GSR), Electroencephalogram (EEG) and Heart Rate (HR). The user is shown different patterns of tiles lighting up and is asked to replay the pattern back pressing the tiles in the same sequence they lit up. The task difficulty depends on the length of the observed pattern (3-6 keys). This task is designed to explore the effect of cognitive load on physiological cues, and if pupil dilation, EEG, GSR and HR can be used as measures of cognitive load.

In a remote collaboration involving a physical task, visualising gaze behaviours may compensate for other unavailable communication channels. In this paper, we report on a 360° panoramic Mixed Reality (MR) remote collaboration system that shares gaze behaviour visualisations between a local user in Augmented Reality and a remote collaborator in Virtual Reality. We conducted two user studies to evaluate the design of MR gaze interfaces and the effect of gaze behaviour (on/off) and gaze style (bi-/uni-directional). The results indicate that gaze visualisations amplify meaningful joint attention and improve co-presence compared to a no gaze condition. Gaze behaviour visualisations enable communication to be less verbally complex therefore lowering collaborators' cognitive load while improving mutual understanding. Users felt that bi-directional behaviour visualisation, showing both collaborator's gaze state, was the preferred condition since it enabled easy identification of shared interests and task progress.

When collaborating face-to-face, people commonly use the surfaces and spaces around them to perform sensemaking tasks, such as spatially organising documents, notes or images. However, when people collaborate remotely using desktop interfaces they no longer feel like they are sharing the same space. This limitation may be overcome through collaboration in immersive environments, which simulate the physical in-person experience. In this paper, we report on a between-groups study comparing collaborations on image organisation tasks, in an immersive Virtual Reality (VR) environment to more conventional desktop conferencing. Collecting data from 40 subjects in groups of four, we measured task performance, user behaviours, collaboration engagement and awareness. Overall, the VR and desktop interface resulted in similar speed, accuracy and social presence rating, but we observed more conversations and interaction with objects, and more equal contributions to the interaction from participants within groups in VR. We also identified differences in coordination and collaborative awareness behaviours between VR and desktop platforms. We report on a set of systematic measures for assessing VR collaborative experience and a new analysis tool that we have developed to capture user behaviours in collaborative setting. Finally, we provide design considerations and directions for future work.

This paper investigates the use of immersive virtual reconstructions as an aid for jurors during a courtroom trial. The findings of a between-participant user study on memory and decision-making are presented in the context of viewing a simulated hit-run-death scenario. Participants listened to the opening statement of a prosecutor and a defence attorney before viewing the crime scene in Virtual Reality (VR) or as still images. We compare the effects on cognition and usability of using VR over images presented on a screen. We found several significant improvements, including that VR led to more consistent decision-making among participants. This shows that VR could provide a promising solution for the court to present crime scenes when site visitations are not possible.

Empathic Computing is an emerging research field that aims to use technology to create deeper shared understanding or empathy between people [1]. The field sits at the junction of research in Natural Collaboration, Experience Capture and Implicit Understanding. Technologies such as Augmented Reality (AR) and Virtual Reality (VR), can be combined with the sensing of human physiological signals to create new types of collaborative experiences. For example, Empathy Glasses [2] use gaze- and face-tracking to share non-verbal communication cues and enhance remote collaborative. More complex tools, such as EEG, can measure brain activity synchronization and physiological states not normally perceived by humans [3].

Many studies have focused on Virtual Reality (VR) frameworks for remotely controlling robotic systems. Although VR systems have been used to teleoperate robots in simple scenarios, their effectiveness in terms of accuracy, speed, and usability has not been rigorously evaluated for complex tasks that require accurate trajectories. In this work, an Enhanced Virtual Reality (EVR) framework for robotic teleoperation is evaluated to assess if it can be efficiently used in complex tasks that require accurate control of the robotic end-effector. The environment and the employed robot are captured using RGB-D cameras, while the remote user controls the motion of the robot with VR controllers. The captured data are transmitted and reconstructed in 3D so as to allow the remote user to monitor the task execution progress in real time, using a VR headset. The EVR system is compared with two other interface alternatives: i) teleoperation in pure VR (the model of the robot is rendered with respect to its real joint states), and ii) teleoperation in EVRR (the model of the robot is superimposed on the real robot). The results show that pure point cloud interfaces suffer from visualization issues, reducing the effectiveness of the robot teleoperation. However, the accuracy and user experience can be greatly improved by including the robot model.

In this paper, we present a novel collaboration tool, OmniGlobeVR, which is an asymmetric system that supports communication and collaboration between a VR user (occupant) and multiple non-VR users (designers) across the virtual and physical platform. OmniGlobeVR allows designer(s) to explore the VR space from any point of view using two view modes: a 360° first-person mode and a third-person mode. In addition, a shared gaze awareness cue is provided to further enhance communication between the occupant and the designer(s). Finally, the system has a face window feature that allows designer(s) to share their facial expressions and upper body view with the occupant for exchanging and expressing information using nonverbal cues. We conducted a user study to evaluate the OmniGlobeVR, comparing three conditions: (1) first-person mode with the face window, (2) first-person mode with a solid window, and (3) third-person mode with the face window. We found that the first-person mode with the face window required significantly less mental effort, and provided better spatial presence, usability, and understanding of the partner's focus. We discuss the design implications of these results and directions for future research.

Supporting natural communication cues is critical for people to work together remotely and face-to-face. In this paper we present a Mixed Reality (MR) remote collaboration system that enables a local worker to share a live 3D panorama of his/her surroundings with a remote expert. The remote expert can also share task instructions back to the local worker using visual cues in addition to verbal communication. We conducted a user study to investigate how sharing augmented gaze and gesture cues from the remote expert to the local worker could affect the overall collaboration performance and user experience. We found that by combing gaze and gesture cues, our remote collaboration system could provide a significantly stronger sense of co-presence for both the local and remote users than using the gaze cue alone. The combined cues were also rated significantly higher than the gaze in terms of ease of conveying spatial actions.

Speech Emotion Recognition (SER) has been a challenging task on which researchers have been working for decades. Recently, Deep Learning (DL) based approaches have been shown to perform well in SER tasks; however, it has been noticed that their superior performance is limited to the distribution of the data used to train the model. In this paper, we present an analysis of using autoencoders to improve the generalisability of DL based SER solutions. We train a sparse autoencoder using a large speech corpus extracted from social media. Later, the trained encoder part of the autoencoder is reused as the input to a long short-term memory (LSTM) network, and the encoder-LSTM modal is re-trained on an aggregation of five commonly used speech emotion corpora. Our evaluation uses an unseen corpus in the training & validation stages to simulate 'in the wild' condition and analyse the generalisability of our solution. A performance comparison is carried out between the encoder based model and a model trained without an encoder. Our results show that the autoencoder based model improves the unweighted accuracy of the unseen corpus by 8%, indicating autoencoder based pre-training can improve the generalisability of DL based SER solutions.

Presence, the feeling of being there, is an important factor that affects the overall experience of Virtual Reality (VR). Higher presence commonly provides a better experience in VR than lower presence. However, presence is commonly measured subjectively through postexperience questionnaires, which can suffer from participant biases, dishonest answers, and fatigue. It can also be difficult for subjects to accurately remember their feelings of presence after they have left the VR experience. In this paper, we measured the effects of different levels of presence (high and low) in VR using physiological and neurological signals. The experiment involved 24 participants in a between-subjects design. Results indicated a significant effect of presence on both physiological and neurological signals. We noticed that higher presence results in higher heart rate, less visual stress, higher theta and beta activities in the frontal region, and higher alpha activities in the parietal region. These findings and insights could lead to an alternative objective measure of presence.

This research investigates enhancing first-person view (FPV) task instruction videos by applying Augmented Reality (AR) visualisation of spatial cues. With personal mobile devices, recording and sharing a video clip has become very easy, and how-to videos are becoming popular on social video sharing services. Instructional videos are actively used not only in formal education and training, but also in everyday life. However, video clips are limited to two-dimensional representation of the task space, making it hard for the viewer to follow and match the objects in the video to those in the real world task space. We propose augmenting task instruction videos with AR visualisation of spatial cues to overcome this problem, focusing on creating and viewing FPV instruction videos. We designed and implemented a prototype system, AR Tips, which allows users to capture and share augmented FPV instruction videos on a wearable AR device. We conducted a user study to evaluate the benefit of our approach, and the results showed that with the help of augmented spatial cues users better understood the instructions, performed the tasks faster with fewer errors, and had lower mental effort.

This paper outlines the objectives of the working group on developing a model Augmented Reality curriculum for higher education. We motivate the need for the model curriculum by the growing Augmented Reality industry and subsequent demand for trained professionals. While the industry is growing, the educational offers that train the required skills remain limited and fragmented. The working group will address this challenge by surveying the state of the art in Augmented Reality education are reviewing available data on industry requirements. Based on the results, the group will develop a new model Augmented Reality curriculum. The working group will also develop future work recommendations for the design of teaching materials and integration of Augmented Reality in computing curricula.

We have proposed a proof of concept of a gaze window interface. A gaze window interface uses the user's gaze point to show the relevant content nearby the current gaze point. A system of a gaze window interface has been implemented that allows users to manage looking at multiple objects of interest on the screen while the system is responsive to the user's gaze points with relevant information shown in the gaze window. We also conducted user studies to investigate the effects of the interface on user performance and behaviors. In this paper, we report on a user study in which the gaze window was compared with a mouse window for supporting single user data entry tasks. We describe details of the study design and conduction and present the results.

With the advancement of Artificial Intelligence technology to make smart devices, understanding how humans develop trust in virtual agents is emerging as a critical research field. Through our research, we report on a novel methodology to investigate user’s trust in auditory assistance in a Virtual Reality (VR) based search task, under both high and low cognitive load and under varying levels of agent accuracy. We collected physiological sensor data such as electroencephalography (EEG), galvanic skin response (GSR), and heart-rate variability (HRV), subjective data through questionnaire such as System Trust Scale (STS), Subjective Mental Effort Questionnaire (SMEQ) and NASA-TLX. We also collected a behavioral measure of trust (congruency of users’ head motion in response to valid/ invalid verbal advice from the agent). Our results indicate that our custom VR environment enables researchers to measure and understand human trust in virtual agents using the matrices, and both cognitive load and agent accuracy play an important role in trust formation. We discuss the implications of the research and directions for future work.

We present AffectivelyVR, a personalized real-time emotion recognition system in Virtual Reality (VR) that enables an emotion-adaptive virtual environment. We used off-the-shelf Electroencephalogram (EEG) and Galvanic Skin Response (GSR) physiological sensors to train user-specific machine learning models while exposing users to affective 360° VR videos. Since emotions are largely dependent on interpersonal experiences and expressed in different ways for different people, we personalize the model instead of generalizing it. By doing this, we achieved an emotion recognition rate of 96.5% using the personalized KNN algorithm, and 83.7% using the generalized SVM algorithm.

In this work, a Mixed Reality (MR) system is evaluated to assess whether it can be efficiently used in teleoperation tasks that require an accurate control of the robot end-effector. The robot and its local environment are captured using multiple RGB-D cameras, and a remote user controls the robot arm motion through Virtual Reality (VR) controllers. The captured data is streamed through the network and reconstructed in 3D, allowing the remote user to monitor the state of execution in real time through a VR headset. We compared our method with two other interfaces: i) teleoperation in pure VR, with the robot model rendered with the real joint states, and ii) teleoperation in MR, with the rendered model of the robot superimposed on the actual point cloud data. Preliminary results indicate that the virtual robot visualization is better than the pure point cloud for accurate teleoperation of a robot arm.

Research on Augmented Reality (AR)/Mixed Reality (MR) remote collaboration for physical tasks remains a compelling and dynamic area of study. AR systems have been developed which transmit virtual annotations between remote collaborators, but there has been little research on how haptic feedback can also be shared. In this paper, we present a Virtual Reality (VR)-Spatial Augmented Reality (SAR) remote collaborative system that provides haptic feedback with tangible interaction between a local worker and a remote expert helper. Using this system, we conducted a within-subject user study to compare two interfaces for remote collaboration between a local worker and expert helper, one with mid-air free drawing (MFD) and one with tangible physical drawing (TPD). The results showed that there were no significant differences with respect to performance time and operation errors. However, users felt that the TPD interface supporting passive haptic feedback could significantly improve the remote experts’ user experience in VR. Our research provides useful information on the way for gesture- and gaze-based multimodal interaction supporting haptic feedback in AR/MR remote collaboration on physical tasks.

Abstract This paper investigates the effect of using augmented reality (AR) annotations and two different gaze visualizations, head pointer (HP) and eye gaze (EG), in an AR system for remote collaboration on physical tasks. First, we developed a spatial AR remote collaboration platform that supports sharing the remote expert’s HP or EG cues. Then the prototype system was evaluated with a user study comparing three conditions for sharing non-verbal cues: (1) a cursor pointer (CP), (2) HP and (3) EG with respect to task performance, workload assessment and user experience. We found that there was a clear difference between these three conditions in the performance time but no significant difference between the HP and EG conditions. When considering the perceived collaboration quality, the HP/EG interface was statistically significantly higher than the CP interface, but there was no significant difference for workload assessment between these three conditions. We used low-cost head tracking for the HP cue and found that this served as an effective referential pointer. This implies that in some circumstances, HP could be a good proxy for EG in remote collaboration. Head pointing is more accessible and cheaper to use than more expensive eye-tracking hardware and paves the way for multi-modal interaction based on HP and gesture in AR remote collaboration.

research-article Share on Special Issue on Highlights of ACM Intelligent User Interface (IUI) 2018 Authors: Mark Billinghurst School of ITMS, University of South Australia, Adelaide, South Australia, Australia School of ITMS, University of South Australia, Adelaide, South Australia, AustraliaView Profile , Margaret Burnett School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USAView Profile , Aaron Quigley School of Computer Science, University of St. Andrews, St. Andrews, Scotland, United Kingdom School of Computer Science, University of St. Andrews, St. Andrews, Scotland, United KingdomView Profile Authors Info & Claims ACM Transactions on Interactive Intelligent SystemsVolume 10Issue 1March 2020 Article No.: 1pp 1–3https://doi.org/10.1145/3357206Published:12 October 2019Publication History 0citation192DownloadsMetricsTotal Citations0Total Downloads192Last 12 Months27Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In this paper we present ZenG, a neurofeedback ARapplication concept based on Zen Gardening to fostercreativity, self-awareness, and relaxation through embodiedinteractions in a mixed reality environment. We developedan initial prototype which combined physiological sensingthrough EEG with AR visualisation on the Magic LeapDisplay. We evaluated the prototype through preliminaryuser testing with 12 adults. Results suggest users found theexperience to be enjoyable and relaxing, however theapplication could be improved by including more featuresand functionality. ZenG shows the potential for AR toprovide immersive and interactive environments that couldpromote creativity and relaxation, providing solid groundsfor further research.

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

We propose a multi-scale Mixed Reality (MR) collaboration between the Giant, a local Augmented Reality user, and the Miniature, a remote Virtual Reality user, in Giant-Miniature Collaboration (GMC). The Miniature is immersed in a 360-video shared by the Giant who can physically manipulate the Miniature through a tangible interface, a combined 360-camera with a 6 DOF tracker. We implemented a prototype system as a proof of concept and conducted a user study (n=24) comprising of four parts comparing: A) two types of virtual representations, B) three levels of Miniature control, C) three levels of 360-video view dependencies, and D) four 360-camera placement positions on the Giant. The results show users prefer a shoulder mounted camera view, while a view frustum with a complimentary avatar is a good visualization for the Miniature virtual representation. From the results, we give design recommendations and demonstrate an example Giant-Miniature Interaction.

When engaged in communication, people often rely on pointing gestures to refer to out-of-reach content. However, observers frequently misinterpret the target of a pointing gesture. Previous research suggests that to perform a pointing gesture, people place the index finger on or close to a line connecting the eye to the referent, while observers interpret pointing gestures by extrapolating the referent using a vector defined by the arm and index finger. In this paper we present Warping Deixis, a novel approach to improving the perception of pointing gestures and facilitate communication in collaborative Extended Reality environments. By warping the virtual representation of the pointing individual, we are able to match the pointing expression to the observer's perception. We evaluated our approach in a co-located side by side virtual reality scenario. Results suggest that our approach is effective in improving the interpretation of pointing gestures in shared virtual environments.

Remote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D virtual reconstruction of their surroundings. However, each of these techniques has benefits and drawbacks. In this paper we explore mixing 360 video and 3D reconstruction together for remote collaboration, by preserving benefits of both systems while reducing drawbacks of each. We developed a hybrid prototype and conducted user study to compare benefits and problems of using 360 or 3D alone to clarify the needs for mixing the two, and also to evaluate the prototype system. We found participants performed significantly better on collaborative search tasks in 360 and felt higher social presence, yet 3D also showed potential to complement. Participant feedback collected after trying our hybrid system provided directions for improvement.

We have explored the effects of sharing manipulated heart rate feedback in collaborative virtual environments. In our study, we created two types of different virtual environments (active and passive) with different levels of interactions and provided three levels of manipulated heart rate feedback (decreased, unchanged, and increased). We measured the effects of manipulated feedback on Social Presence, affect, physical heart rate, and overall experience. We noticed a significant effect of the manipulated heart rate feedback in affecting scariness and nervousness. The perception of the collaborator's valance and arousal was also affected where increased heart rate feedback perceived as a higher valance and lower arousal. Increased heart rate feedback decreased the real heart rate. The type of virtual environments had a significant effect on social presence, heart rate, and affect where the active environment had better performances across these measurements. We discuss the implications of this and directions for future research.

This work presents a study that explores the differences between authoring Scenario-Based Training (SBT) simulation content using Augmented Reality (AR) and a Desktop interface. Through an iterative design process two interface conditions were developed and then evaluated qualitatively and quantitatively. Our conceptual model is a graph based visualization that is presented to help designers understand the scenario learning artifacts and relationships. Our major contribution relies on the comparison made between the two authoring tools (AR, Desktop) with the same capabilities. Results show that no significant difference was found in time taken to complete tasks nor on the perceived usability of the systems. However, as expected the Desktop interface was perceived as more efficient. Based on these findings, insights on future directions for building AR immersive authoring tools are provided.

In this paper, we explore the effect of showing a remote partner close to user gaze point in a teleconferencing system. We implemented a gaze following function in a teleconferencing system and investigate if this improves the user's feeling of emotional interdependence. We developed a prototype system that shows a remote partner close to the user's current gaze point and conducted a user study comparing it to a condition displaying the partner fixed in the corner of a screen. Our results showed that showing a partner close to their gaze point helped users feel a higher level of emotional interdependence. In addition, we compared the effect of our method between small and big displays, but there was no significant difference in the users' feeling of emotional interdependence even though the big display was preferred.

The decisions made by an Air Attack Supervisor (AAS) helicopter co-pilots in aerial firefighting have critical and immediate impacts. It is difficult to always make fast, high quality decisions due to the mental and physical stress being experienced. Real world training exercises have limitations such as safety, cost, time and difficulty in reproducing events, making frequent training infeasible. Virtual Reality (VR) offers new training opportunities, but it is challenging to create a virtual environment with the analogous level of stress experienced in the real-world. In this paper, we investigate the use of a multi-user, collaborative, multi-sensory (vision, audio, tactile) VR system to produce a realistic training environment for practising aerial firefighting training scenarios. We focus on a comparison between our VR training system, an equivalent real-world field training and an existing radio-only exercise currently in use, where we compare Heart-Rate Variability (HRV) and self reported stress using the Short Stress State Questionnaire (SSSQ). We conducted the study with real trainee AAS firefighters to determine the effectiveness of the system. Our results show that there were no significant differences between the VR training exercise and the real-world exercise in terms of the level of stress, measured by HRV, and no significant difference between VR and radio-only exercises, as reported by the SSSQ.

Virtual Reality (VR) is effective in various training scenarios across multiple domains, such as education, health and defense. However, most of those applications are not adaptive to the real-time cognitive or subjectively experienced load placed on the trainee. In this paper, we explore a cognitively adaptive training system based on real-time measurement of task related alpha activity in the brain. This measurement was made by a 32-channel mobile Electroencephalography (EEG) system, and was used to adapt the task difficulty to an ideal level which challenged our participants, and thus theoretically induces the best level of performance gains as a result of training. Our system required participants to select target objects in VR and the complexity of the task adapted to the alpha activity in the brain. A total of 14 participants undertook our training and completed 20 levels of increasing complexity. Our study identified significant differences in brain activity in response to increasing levels of task complexity, but response time did not alter as a function of task difficulty. Collectively, we interpret this to indicate the brain's ability to compensate for higher task load without affecting behaviourally measured visuomotor performance.

Research has shown that Mixed Presence Groupware (MPG) systems are a valuable collaboration tool. However research into MPG systems is limited to a handful of tabletop and Virtual Reality (VR) systems with no exploration of Head-Mounted Display (HMD) based Augmented Reality (AR) solutions. We present a new system with two local users and one remote user using HMD based AR interfaces. Our system provides tools allowing users to layout a room with the help of a remote user. The remote user has access to a marker and pointer tools to assist in directing the local users. Feedback collected from several groups of users showed that our system is easy to learn but could have increased accuracy and consistency.

We propose enhancing live 360 panorama-based Mixed Reality (MR) remote collaboration through supporting visual annotation cues. Prior work on live 360 panorama-based collaboration used MR visualization to overlay visual cues, such as view frames and virtual hands, yet they were not registered onto the shared physical workspace, hence had limitations in accuracy for pointing or marking objects. Our prototype system uses spatial mapping and tracking feature of an Augmented Reality head-mounted display to show visual annotation cues accurately registered onto the physical environment. We describe the design and implementation details of our prototype system, and discuss on how such feature could help improve MR remote collaboration.

This paper investigates how two different unique gaze visualizations (the head pointer(HP), eye gaze(EG)) affect table-size physical tasks in Mixed Reality (MR) remote collaboration. We developed a remote collaborative MR Platform which supports sharing of the remote expert's HP and EG. The prototype was evaluated with a user study comparing two conditions: sharing HP and EG with respect to their effectiveness in the performance and quality of cooperation. There was a statistically significant difference between two conditions on the performance time, and HP is a good proxy for EG in remote collaboration.

This paper investigates the effect of avatar appearance on Social Presence and users' perception in an Augmented Reality (AR) telep-resence system. Despite the development of various commercial 3D telepresence systems, there has been little evaluation and discussions about the appearance of the collaborator's avatars. We conducted two user studies comparing the effect of avatar appearances with three levels of body part visibility (head & hands, upper body, and whole body) and two different character styles (realistic and cartoon-like) on Social Presence while performing two different remote collaboration tasks. We found that a realistic whole body avatar was perceived as being the best for remote collaboration, but an upper body or cartoon style could be considered as a substitute depending on the collaboration context. We discuss these results and suggest guidelines for designing future avatar-mediated AR remote collaboration systems.

Modern volumetric projection-based telepresence approaches are capable of providing realistic full-size virtual representations of remote people. Interacting with full-size people may not be desirable due to the spatial constraints of the physical environment, application context, or display technology. However, the miniaturization of remote people is known to create an eye gaze matching problem. Eye-contact is essential to communication as it allows for people to use natural nonverbal cues and improves the sense of “being there”. In this paper we discuss the design space for interacting with volumetric representations of people and present an approach for dynamically manipulating scale, orientation and the position of holograms which guarantees eye-contact. We created a working augmented reality-based prototype and validated it with 14 participants.

Human trust is a psycho-physiological state that is difficult to measure, yet is becoming increasingly important for the design of human-computer interactions. This paper explores if human trust can be measured using physiological measures when interacting with a computer interface, and how it correlates with cognitive load. In this work, we present a pilot study in Virtual Reality (VR) that uses a multi-sensory approach of Electroencephalography (EEG), galvanic skin response (GSR), and Heart Rate Variability (HRV) to measure trust with a virtual agent and explore the correlation between trust and cognitive load. The goal of this study is twofold; 1) to determine the relationship between biosignals, or physiological signals with trust and cognitive load, and 2) to introduce a pilot study in VR based on cognitive load level to evaluate trust. Even though we could not report any significant main effect or interaction of cognitive load and trust from the physiological signal, we found that in low cognitive load tasks, EEG alpha band power reflects trustworthiness on the agent. Moreover, cognitive load of the user decreases when the agent is accurate regardless of task’s cognitive load. This could be possible because of small sample size, tasks not stressful enough to induce high cognitive load due to lab study and comfortable environment or timestamp synchronisation error due to fusing data from various physiological sensors with different sample rate.

Augmented Reality (AR) mirrors can show virtual objects overlaid onto the physical world reflected in the mirror. Optical-reflection type AR mirror displays use half-silvered mirrors attached in front of a digital display. However, prior work suffered from visual depth mismatch between the optical reflection of the 3D physical space and 2D images displayed on the surface of the mirror. In this research, we use 3D visualisation to overcome this problem and improve the user experience by providing better depth perception for watching and interacting with the content displayed on an AR mirror. As a proof of concept, we developed two prototype optical-reflection type 3D AR mirror displays, one using glasses-free multi-view 3D display and another using a head tracked 3D stereoscopic display that supports hand gesture interaction.

Mixed Reality (MR) remote collaboration provides an enhanced immersive experience where a remote user can provide verbal and nonverbal assistance to a local user to increase the efficiency and performance of the collaboration. This is usually achieved by sharing the local user's environment through live 360 video or a 3D scene, and using visual cues to gesture or point at real objects allowing for better understanding and collaborative task performance. While most of prior work used one of the methods to capture the surrounding environment, there may be situations where users have to choose between using 360 panoramas or 3D scene reconstruction to collaborate, as each have unique benefits and limitations. In this paper we designed a prototype system that combines 360 panoramas into a 3D scene to introduce a novel way for users to interact and collaborate with each other. We evaluated the prototype through a user study which compared the usability and performance of our proposed approach to live 360 video collaborative system, and we found that participants enjoyed using different ways to access the local user's environment although it took them longer time to learn to use our system. We also collected subjective feedback for future improvements and provide directions for future research.

Head and eye movement can be leveraged to improve the user's interaction repertoire for wearable displays. Head movements are deliberate and accurate, and provide the current state-of-the-art pointing technique. Eye gaze can potentially be faster and more ergonomic, but suffers from low accuracy due to calibration errors and drift of wearable eye-tracking sensors. This work investigates precise, multimodal selection techniques using head motion and eye gaze. A comparison of speed and pointing accuracy reveals the relative merits of each method, including the achievable target size for robust selection. We demonstrate and discuss example applications for augmented reality, including compact menus with deep structure, and a proof-of-concept method for on-line correction of calibration drift.

We present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user's gaze direction and body gestures while it transforms in size and orientation to stay within the AR user's field of view. A user study was conducted to evaluate Mini-Me in two collaborative scenarios: an asymmetric remote expert in VR assisting a local worker in AR, and a symmetric collaboration in urban planning. We found that the presence of the Mini-Me significantly improved Social Presence and the overall experience of MR collaboration.

It has been observed that a robot shown as suffering is enough to cause an empathic response from a person. Whether the response is a fleeting reaction with no consequences or a meaningful perspective change with associated behavior modifications is not clear. Existing work has been limited to measurements made at the end of empathy inducing experimental trials rather measurements made over time to capture consequential behavioral pattern. We report on preliminary results collected from a study that attempts to measure how the actions of a participant may be altered by empathy for a robot companion. Our findings suggest that induced empathy can in fact have a significant impact on a person's behavior to the extent that the ability to fulfill a mission may be affected.

Collaboration is an important application area for virtual reality (VR). However, unlike in the real world, collaboration in VR misses important empathetic cues that can make collaborators aware of each other’s emotional states. Providing physiological feedback, such as heart rate or respiration rate, to users in VR has been shown to create a positive impact in single user environments. In this paper, through a rigorous mixed-factorial user experiment, we evaluated how providing heart rate feedback to collaborators influences their collaboration in three different environments requiring different kinds of collaboration. We have found that when provided with real-time heart rate feedback participants felt the presence of the collaborator more and felt that they understood their collaborator’s emotional state more. Heart rate feedback also made participants feel more dominant when performing the task. We discuss the implication of this research for collaborative VR environments, provide design guidelines, and directions for future research.

Sharing and watching live 360 panorama video is available on modern social networking platforms, yet the communication is often a passive one-directional experience. This research investigates how to further improve live 360 panorama based remote collaborative experiences by adding Mixed Reality (MR) cues. SharedSphere is a wearable MR remote collaboration system that enriches a live captured immersive panorama based collaboration through MR visualisation of non-verbal communication cues (e.g., view awareness and gestures cues). We describe the design and implementation details of the prototype system, and report on a user study investigating how MR live panorama sharing affects the user's collaborative experience. The results showed that providing view independence through sharing live panorama enhances co-presence in collaboration, and the MR cues help users understanding each other. Based on the study results we discuss design implications and future research direction.

Strong empathic bonding between members of a team can elevate team performance tremendously but it is not clear how such bonding within human-machine teams may impact upon mission success. Prior work using self-reporting surveys and end-of-task metrics do not capture how such bonding may evolve over time and impact upon task fulfillment. Furthermore, sensor-based measures do not scale easily to facilitate the need to collect substantial data for measuring potentially subtle effects. We introduce TEAMMATE, a system designed to provide insights into the emotional dynamics humans may form for machine teammates that could critically impact upon the design of human machine teams.

Situation Awareness (SA) is an essential skill in Air Attack Supervision (AAS) for aerial based wildfire firefighting. The display types used for Virtual Reality Training Systems (VRTS) afford different visual SA depending on the Field of View (FoV) as well as the sense of presence users can obtain in the virtual environment. We conducted a study with 36 participants to evaluate SA acquisition in three display types: a high-definition TV (HDTV), an Oculus Rift Head-Mounted Display (HMD) and a 270° cylindrical simulation projection display called the SimPit. We found a significant difference between the HMD and the HDTV, as well as with the SimPit and the HDTV for the three levels of SA.

In 2008, Zhou et al. presented a survey paper summarizing the previous ten years of ISMAR publications, which provided invaluable insights into the research challenges and trends associated with that time period. Ten years later, we review the research that has been presented at ISMAR conferences since the survey of Zhou et al., at a time when both academia and the AR industry are enjoying dramatic technological changes. Here we consider the research results and trends of the last decade of ISMAR by carefully reviewing the ISMAR publications from the period of 2008-2017, in the context of the first ten years. The numbers of papers for different research topics and their impacts by citations were analyzed while reviewing them-which reveals that there is a sharp increase in AR evaluation and rendering research. Based on this review we offer some observations related to potential future research areas or trends, which could be helpful to AR researchers and industry members looking ahead.

The advancements in Mixed Reality (MR), Unmanned Aerial Vehicle, and multi-scale collaborative virtual environments have led to new interface opportunities for remote collaboration. This paper explores a novel concept of flying telepresence for multi-scale mixed reality remote collaboration. This work could enable remote collaboration at a larger scale such as building construction. We conducted a user study with three experiments. The first experiment compared two interfaces, static and dynamic IPD, on simulator sickness and body size perception. The second experiment tested the user perception of a virtual object size under three levels of IPD and movement gain manipulation with a fixed eye height in a virtual environment having reduced or rich visual cues. Our last experiment investigated the participant's body size perception for two levels of manipulation of the IPDs and heights using stereo video footage to simulate a flying telepresence experience. The studies found that manipulating IPDs and eye height influenced the user's size perception. We present our findings and share the recommendations for designing a multi-scale MR flying telepresence interface.

This paper showcases one way of how virtual reconstruction can be used in a courtroom. The results of a pilot study on narrative and spatial memory are presented in the context of viewing real and virtual copies of a simulated crime scene. Based on current court procedures, three different viewing options were compared: photographs, a real life visit, and a 3D virtual reconstruction of the scene viewed in a Virtual Reality headset. Participants were also given a written narrative that included the spatial locations of stolen goods and were measured on their ability to recall and understand these spatial relationships of those stolen items. The results suggest that Virtual Reality is more reliable for spatial memory compared to photographs and that Virtual Reality provides a compromise for when physical viewing of crime scenes are not possible. We conclude that Virtual Reality is a promising medium for the court.

Previous research has demonstrated that Augmented Reality can reduce a user's task response time and mental effort when completing a procedural task. This paper investigates techniques to improve user performance and reduce mental effort by providing projector-based Spatial Augmented Reality predictive cues for future responses. The objective of the two experiments conducted in this study was to isolate the performance and mental effort differences from several different annotation cueing techniques for simple (Experiment 1) and complex (Experiment 2) button-pressing tasks. Comporting with existing cognitive neuroscience literature on prediction, attentional orienting, and interference, we hypothesized that for both simple procedural tasks and complex search-based tasks, having a visual cue guiding to the next task's location would positively impact performance relative to a baseline, no-cue condition. Additionally, we predicted that direction-based cues would provide a more significant positive impact than target-based cues. The results indicated that providing a line to the next task was the most effective technique for improving the users' task time and mental effort in both the simple and complex tasks.

Interfaces for collaborative tasks, such as multiplayer games can enable more effective and enjoyable collaboration. However, in these systems, the emotional states of the users are often not communicated properly due to their remoteness from one another. In this paper, we investigate the effects of showing emotional states of one collaborator to the other during an immersive Virtual Reality (VR) gameplay experience. We created two collaborative immersive VR games that display the real-time heart-rate of one player to the other. The two different games elicited different emotions, one joyous and the other scary. We tested the effects of visualizing heart-rate feedback in comparison with conditions where such a feedback was absent. The games had significant main effects on the overall emotional experience.

Science fiction media has had a long lasting influence on the progression of interactive technology, however recently contradictions are emerging in the development of the two disciplines. Therefore, in this exploratory position paper we report on the insights attained through a day long workshop amongst scientists and researchers on how the collaboration between science fiction and Human Computer Interaction (HCI) can be advanced. Discussions in the workshop focused on detailing the relationship between HCI and science fiction. In conclusion, as our main contribution an action plan and agenda is presented for facilitating deeper influences amongst the two disciplines.

Using game balancing techniques can provide the right level of challenge and hence enhance player engagement for sport players with different skill levels. Digital technology can support and enhance balancing techniques in sports, for example, by adjusting players' level of intensity based on their heart rate. However, there is limited knowledge on how to design such balancing and its impact on the user experience. To address this we created two novel balancing techniques enabled by digitally augmenting a table tennis table. We adjusted the more skilled player's performance by inducing two different styles of play and studied the effects on game balancing and player engagement. We showed that by altering the more skilled player's performance we can balance the game through: (i) encouraging game mistakes, and (ii) changing the style of play to one that is easier for the opponent to counteract. We outline the advantages and disadvantages of each approach, extending our understanding of game balancing design. We also show that digitally augmenting sports offers opportunities for novel balancing techniques while facilitating engaging experiences, guiding those interested in HCI and sports.

We present results from research exploring the effect of sharing virtual gaze and pointing cues in a wearable interface for remote collaboration. A local worker wears a Head-mounted Camera, Eye-tracking camera and a Head-Mounted Display and shares video and virtual gaze information with a remote helper. The remote helper can provide feedback using a virtual pointer on the live video view. The prototype system was evaluated with a formal user study. Comparing four conditions, (1) NONE (no cue), (2) POINTER, (3) EYE-TRACKER and (4) BOTH (both pointer and eye-tracker cues), we observed that the task completion performance was best in the BOTH condition with a significant difference of POINTER and EYETRACKER individually. The use of eye-tracking and a pointer also significantly improved the co-presence felt between the users. We discuss the implications of this research and the limitations of the developed system that could be improved in further work.

Early research on head-worn computers (HWCs) has focused on hardware and specific applications. However, there is little research about the everyday usage of head-worn computers in particular aspects such as: context of use, social acceptance across different activities, audiences and interaction techniques. This paper provides insights into the use of head-worn computers by capturing the opinions of novice and expert users through a survey, a three-week diary study, and interviews. The overarching finding is that the context of use is critical, either due to the need to support micro-interactions, or because the interaction paradigm itself should depend on the context of use.

This half-day hands-on studio will teach how to design and develop effective interfaces for head mounted and wrist worn wearable computers through the application of user-centered design principles. Attendees will learn gain the knowledge and tools needed to rapidly develop prototype applications, and also complete a hands-on design task. They will also learn good design guidelines for wearable systems and how to apply those guidelines. A variety of tools will be used that do not require any hardware or software experience, many of which are free and/or open source. Attendees will also be provided with material that they can use to continue their learning after the studio is over.

Although many progresses have been accomplished in multimodal interaction, most researchers still treat each modality such as vision and speech, separately. They integrate the results at the application stage. This is because the roles of multiple modalities and their interactions continue to be quantified and precisely understood. However, there are many remaining issues in combining each modality individually. This paper will highlight the main vision problems based on our review for multimodal applications. This review paper will give an overview of the Augmented Reality (AR) technologies which are contributing in most of recent multimodal applications. We cluster vision techniques according to the natural human senses such as face, gesture, and speech that are frequently used in multimodal applications. The main contribution of this paper is to consolidate some of the main issues and approaches in vision-based technique, and to study some of the applications in AR that have been developed within the context of multimodal interaction. We conclude this paper with the future directions.

Augmented Reality (AR) is a technology that allows users to view and interact in real time with virtual images seamlessly superimposed over the real world. AR systems can be used to create unique collaborative experiences. For example, co-located users can see shared 3D virtual objects that they interact with, or a user can annotate the live video view of a remote worker, enabling them to collaborate at a distance. The overall goal is to augment the face-to-face collaborative experience, or to enable remote people to feel that they are virtually co-located. In this special issue on collaboration in augmented reality, we begin with the visions of science fiction authors of future technologies that might significantly improve collaboration, then introduce research articles which describe progress towards these visions, finally we outline a research agenda discussing the work still to be done.

Playing exertion games with others can be engaging. However, players with different physical skill levels competing against each other can experience reduced engagement because they are either not challenged enough, or challenged too much. Balancing methods can address this; however, there is only limited understanding of balancing in exertion games. In this paper, we identify two distinct dimensional balancing techniques: "internal adjustment" and "external adjustment". We report results from a study where we measured player engagement after applying these adjustments to a digital table tennis game and the traditional table tennis game, finding two disengagement factors: "unexpected physical challenges" and "unacceptable competitive advantage". Based on these factors we derived a set of exertion game design considerations. We conclude that applying digital technology to a physical game can change the required skill level to play the game, and this can affect the impact of these adjustments on player engagement. These results enhances our understanding of balancing in exertion games, supporting the benefits of playing exertion games with others.

The course will teach how to create compelling user experiences for wearable computers focusing on design guidelines, prototyping tools, research directions, and a hands-on design experience. These topics will be presented using a number of platforms such as Google Glass, the Recon Jet and Vuzix M-100, although the material will be relevant to other wearable devices. The class will begin with an overview of almost 50 years of wearable computing, beginning with the casino computers of Ed Thorp, through the pioneering efforts of researchers at CMU and MIT, to the most recent commercial systems. The key technology components of a wearable system will be covered, as well as some of the theoretical underpinnings. Next, a set of design guidelines for developing wearable user interfaces will be presented. These include lessons learned from using wearables on a daily basis, design patterns from existing wearable interfaces, and relevant results from the research community. These will be presented in enough details that attendees will be able to use them in their own wearable designs. The third section of the course will introduce a number of tools that can be used for rapid prototyping of wearable interfaces. These include screen-building tools such as Glasssim, through to templating tools that support limited interactivity, and simple programming tools such as Processing. This will lead into a section that discusses the technology of wearable systems in more detail. For example, the different types of head mounted displays for wearables, tracking technology for wearable AR interfaces, input devices, etc. Finally, we will discuss active areas of research that will affect wearable interfaces over the next few years. This includes technologies such as new display hardware, input devices, body worn sensors, and connectivity. The course will have the following educational goals: • Provide an introduction to head mounted wearable computers • Give an understanding of current wearable computing technology • Describe key design principles/interface metaphors • Provide an overview of the relevant human perceptual principles • Explain how to use Processing for rapid prototyping • Show how to capturing and use sensor input • Outline active areas of research in wearable computing • Hands on demonstrations with Google Glass and other wearable computers.

Video conferencing is becoming more widely used in areas other than face-to-face conversation, such as sharing real world experience with remote friends or family. In this paper we explore how adding augmented visual communication cues can improve the experience of sharing remote task space and collaborating together. We developed a prototype system that allows users to share live video view of their task space taken on a Head Mounted Display (HMD) or Handheld Display (HHD), and communicate through not only voice but also using augmented pointer or annotations drawn on the shared view. To explore the effect of having such an interface for remote collaboration, we conducted a user study comparing three video-conferencing conditions with different combination of communication cues: (1) voice only, (2) voice + pointer, and (3) voice + annotation. The participants used our remote collaboration system to share a parallel experience of puzzle solving in the user study, and we found that adding augmented visual cues significantly improved the sense of being together. The pointer was the most preferred additional cue by users for parallel experience, and there were different states of the users' behavior found in remote collaboration.

In this half-day workshop we will explore how Augmented Reality (AR) and Mediated Reality (MR) can be used to develop radically new types of collaborative experiences that overcome some of the limitations of current conferencing systems. In combination, AR and MR technologies could be used to merge the shared perceived realities of different users as well as enriching their own individual experience in a collaborative task. The goal of the workshop is to bring together researchers who are interested developing collaborative systems using AR and MR technologies. They will build a picture of current and prior research on collaboration in AR and MR as well as set up a common research agenda for work going forward. Topics of the workshop will address open research issues and include but are not restricted to the following: •Case studies on using MR/AR for collaboration •Tools for building collaborative MR/AR systems •Effects of MR/AR on trust, presence, and coordination •Interaction models for collaboration in MR/AR •Tools for collaboration in MR/AR •Collaboration awareness in MR/AR.

In order for natural interaction in Augmented Reality (AR) to become widely adopted, the techniques used need to be shown to support precise interaction, and the gestures used proven to be easy to understand and perform. Recent research has explored free-hand gesture interaction with AR interfaces, but there have been few formal evaluations conducted with such systems. In this paper we introduce and evaluate two natural interaction techniques: the free-hand gesture based Grasp-Shell, which provides direct physical manipulation of virtual content; and the multi-modal Gesture-Speech, which combines speech and gesture for indirect natural interaction. These techniques support object selection, 6 degree of freedom movement, uniform scaling, as well as physics-based interaction such as pushing and flinging. We conducted a study evaluating and comparing Grasp-Shell and Gesture-Speech for fundamental manipulation tasks. The results show that Grasp-Shell outperforms Gesture-Speech in both efficiency and user preference for translation and rotation tasks, while Gesture-Speech is better for uniform scaling. They could be good complementary interaction methods in a physics-enabled AR environment, as this combination potentially provides both control and interactivity in one interface. We conclude by discussing implications and future directions of this research.

We demonstrate an Augmented Reality (AR) system that utilizes a combination of direct free hand interaction and indirect multimodal gesture and speech interface. A three-dimensional (3D) design sandbox application, featuring online object creation, has been developed to illustrate the use case of our system that supports dual interaction techniques.

In this paper we describe the concept of Social Panoramas that combine panorama images, Mixed Reality, and wearable computers to support remote collaboration. We have developed a prototype that allows panorama images to be explored in real time between a Google Glass user and a remote tablet user. This uses a variety of cues for supporting awareness, and enabling pointing and drawing. We conducted a study to explore if these cues can increase Social Presence. The results suggest that increased interaction does not increase Social Presence, but tools with a higher perceived usability show an improved sense of presence.

This poster presents an Augmented Reality (AR) system for remote collaboration that allows a remote user to navigate a local user's scene, independently from their viewpoint. This is achieved by using a 3D scan and reconstruction of the user's environment. The remote user can place virtual objects in the scene that the local user views through a head mounted display (HMD), helping them place real objects. A user study tested how the amount of remote view independence affected collaboration. We found that increased view independence led to faster task completion, more user confidence, and a decrease in verbal communication, but object placement accuracy remained unchanged.

Intelligent user interfaces have traditionally been used to create systems that respond intelligently to user input. However there is a recent trend towards Empathic Interfaces that are designed to go beyond understanding user input and to recognize emotional state and user feelings. In this presentation we explore how Augmented Reality (AR) can be used to convey that emotional state and so allow users to capture and share emotional experiences. In this way AR not only overlays virtual imagery on the real world, but also can create deeper understanding of user's experience at particular locations and points in time. The recent emergence of truly wearable systems, such as Google Glass, provide a platform for Empathic Communication using AR. Examples will be shown from research conducted at the HIT Lab NZ and other research organizations, and key areas for future research described.

Augmented Reality (AR) is technology that allows virtual imagery to be seamlessly integrated into the real world. Although first developed in the 1960's it has only been recently that AR has become widely available, through platforms such as the web and mobile phones. However most AR interfaces have very simple interaction, such as using touch on phone screens or camera tracking from real images. New depth sensing and gesture tracking technologies such as Microsoft Kinect or Leap Motion have made is easier than ever before to track hands in space. Combined with speech recognition and AR tracking and viewing software it is possible to create interfaces that allow users to manipulate 3D graphics in space through a natural combination of speech and gesture. In this paper I will review previous research in multimodal AR interfaces and give an overview of the significant research questions that need to be addressed before speech and gesture interaction can become commonplace.

Conventional 2D touch-based interaction methods for handheld Augmented Reality (AR) cannot provide intuitive 3D interaction due to a lack of natural gesture input with real-time depth information. The goal of this research is to develop a natural interaction technique for manipulating virtual objects in 3D space on handheld AR devices. We present a novel method that is based on identifying the positions and movements of the user's fingertips, and mapping these gestures onto corresponding manipulations of the virtual objects in the AR scene. We conducted a user study to evaluate this method by comparing it with a common touch-based interface under different AR scenarios. The results indicate that although our method takes longer time, it is more natural and enjoyable to use.

Welcome to the Twelfth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2013).

In this research we explore how a handheld display (HHD) can be used to provide input into an Augmented Reality (AR) conferencing application shown on a head mounted display (HMD). Although AR has successfully been used for many collaborative applications, there has been little research on using HHD and HMD together to enhance remote conferencing. This research investigates two different HHD interfaces and methods for supporting file sharing in an AR conferencing application. A formal evaluation compared four different conditions and found that an Exocentric view and using Visual cues for requesting content produced the best performance. The results were used to create a set of basic design guidelines for future research and application development.

Most mobile Augmented Reality (AR) systems use either a head mounted display (HMD) or a handheld display (HHD) as a hardware platform. As mobile devices become more affordable, it becomes more common that users own more than one mobile device and use them together. In this research, we investigate Hybrid AR systems that use both HMD and HHD for AR visualization and interaction. In addition to a simple approach of using HMD as a display and HHD as an input device (e.g. a touch pad or a pointer), we further explore novel interaction techniques that can take advantage of having both HMD and HHD closely integrated into one AR system, such as cross-device information sharing, situation adaptive visualization management, and multi-layered visualization.

Mobile Augmented Reality (AR) applications are typically deployed either on head mounted displays (HMD) or handheld displays (HHD). This paper explores novel interaction techniques for a combined HHD-HMD hybrid system that builds on the strengths of each type of device. We use the HMD for viewing AR content and a touch screen HHD for interacting with the content. A prototype system was developed and a user study was conducted comparing four interaction techniques for selection tasks.

We present an Augmented Reality (AR) system where we immerse the user's whole body in the virtual scene using a motion capturing (MoCap) suit. The goal is to allow for seamless interaction with the virtual content within the AR environment. We describe an evaluation study of a prototype application featuring an interactive scenario with a virtual agent. The scenario contains two conditions: in one, the agent has access to the full tracking data of the MoCap suit and therefore is aware of the exact actions of the user, while in the second condition, the agent does not get this information. We then report and discuss the differences we were able to detect regarding the users' perception of the interaction with the agent and give future research directions.

In this research, we explore how different types of augmented gesture communication cues can be used under different view sharing techniques in a remote collaboration system. In a pilot study, we compared four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. Through this study, we found three results. First, users collaborate more efficiently using annotation cues than pointer cues for communicating object position and orientation information. Second, live video becomes more important when quick feedback is needed. Third, the type of gesture cue has more influence on performance and user preference than the type of view sharing method.

In this paper, we describe the KITE, a mobile Augmented Reality (AR) platform that uses a magnetic tracker and a depth sensor for games and interaction development that is typically only available on a desktop system. We have achieved this using off-the-shelf hardware and efficient software that can be easily assembled and executed. We demonstrate four possible modalities based on hand input to provide a platform that game and interaction designers can use to explore new possibilities for gaming in AR.

This paper describes an AR system for remote collaboration using a captured 3D model of the local user's scene. In the system a remote user can manipulate the scene independently of the view of the local user and add AR annotations that appear projected into the real world. Results from a pilot study and the design of a further full study are presented.

The 2013 virtual reality technical achievement award goes to Mark Billinghurst of the HIT Lab NZ at the University of Canterbury, New Zealand, in recognition of technical developments that have significantly advanced research and commercialization in the field of Augmented Reality. Professor Billinghurst has been working in AR for over 15 years and has contributed to significant advancements in the field such co-developing the ARToolKit tracking library, novel interaction metaphors such as Tangible AR, innovative systems such as the MagicBook transitional AR interface, and the first collaborative AR system for mobile phones. His research has been commercialized through four AR companies he has co-founded, and software released into the open source community has been used by hundreds of thousands of developers and millions of end users. His efforts in AR research, commercialization and communication has contributed significantly to the growth of the entire field. The IEEE VGTC is pleased to award Mark Billinghurst the 2013 Virtual Reality Technical Achievement Award.

This article investigates the effect of various design parameters of auditory information display on user performance in two basic information retrieval tasks. We conducted a user test with 22 participants in which sets of sound samples were presented. In the first task, the test participants were asked to detect a given sample among a set of samples. In the second task, the test participants were asked to estimate the relative number of instances of a given sample in two sets of samples. We found that the stimulus onset asynchrony (SOA) of the sound samples had a significant effect on user performance in both tasks. For the sample detection task, the average error rate was about 10% with an SOA of 100 ms. For the numerosity estimation task, an SOA of at least 200 ms was necessary to yield average error rates lower than 30%. Other parameters, including the samples' sound type (synthesized speech or earcons) and spatial quality (multichannel loudspeaker or diotic headphone playback), had no substantial effect on user performance. These results suggest that diotic, or indeed monophonic, playback with appropriately chosen SOA may be sufficient in practical applications for users to perform the given information retrieval tasks, if information about the sample location is not relevant. If location information was provided through spatial playback of the samples, test subjects were able to simultaneously detect and localize a sample with reasonable accuracy.

Presents the recipient of the 2013 Virtual Reality Technical Achievement Award.

The motivation of this workshop is to discuss future direction of content authoring in the field of Augmented Reality, as well as to discuss the current state of art on content creation and content authoring for augmented reality. The workshop will comprise of a paper session where authoring papers, late-breaking results and overviews over state-of-the-art are presented. In the afternoon, we will follow up with discussion sessions on different topics ranging from content creation and authoring to content distribution for AR and a short closing session.

Tradition is alive and well, but there also should be opportunities for innovation at ISMAR.

In this paper, we evaluate a path editing method using a tangible user interface to generate and manipulate the movement path of a 3D object in an Augmented Reality (AR) scene. To generate the movement path, each translation point of a real 3D manipulation prop is examined to determine which point should be used as a control point for the path. Interpolation using splines is then used to reconstruct the path with a smooth line. A dynamic score-based selection method is also used to effectively select small and dense control points of the path. In an experimental evaluation, our method took the same time and generated a similar amount of errors as a more traditional approach, however the number of control points needed was significantly reduced. For control manipulation, the task completion time was quicker and there was less hand movement needed. Our method can be applied to drawing or curve editing methods in AR educational, gaming, and simulation applications.

There is a need for computer aided design tools that support rapid conceptual level design. In this paper we explore and evaluate how intuitive speech and multitouch input can be combined in a multimodal interface for conceptual 3D modeling. Our system, MozArt, is based on a user's innate abilities - speaking and touching, and has a toolbar/button-less interface for creating and interacting with computer graphics models. We briefly cover the hardware and software technology behind MozArt, and present a pilot study comparing our multimodal system with a conventional multitouch modeling interface with first time CAD users. While a larger study is required to obtain statistically significant comparison regarding efficiency and accuracy of the two interfaces, a majority of the participants preferred the multimodal interface over the multitouch. We summarize lessons learned and discuss directions for future research.

The motivation of this workshop is to discuss future directions of content authoring in the field of Augmented Reality, as well as to discuss the current state of art on content creation and asset assembly. The workshop will comprise of a paper session where papers, late-breaking results and overviews over state-of-the-art in content authoring for AR are presented. In the afternoon, we will follow up with a discussion on different topics ranging from AR asset creation to content distribution and a closing session. Our goal is to collect ideas and thoughts of research about desired approaches for authoring content for AR, as well as review current and future needs to achieve a high quality content and at the same time scalable approaches for AR.

Recent advances in mobile computing and augmented reality (AR) technology have lead to popularization of mobile AR applications. Touch screen input is common in mobile devices, and also widely used in mobile AR applications. However, due to unsteady camera view movement, it can be hard to carry out precise interactions in handheld AR environments, for tasks such as tracing physical objects. In this research, we investigate a Snap-To-Feature interaction method that helps users to perform more accurate touch screen interactions by attracting user input points to image features in the AR scene. A user experiment is performed using the method to trace a physical object, which is typical for modeling real objects within the AR scene. The results shows that the Snap-To-Feature method makes a significant difference in the accuracy of touch screen based AR interaction.

Nestor is a real-time recognition and camera pose estimation system for planar shapes. The system allows shapes that carry contextual meanings for humans to be used as Augmented Reality (AR) tracking targets. The user can teach the system new shapes in real time. New shapes can be shown to the system frontally, or they can be automatically rectified according to previously learned shapes. Shapes can be automatically assigned virtual content by classification according to a shape class library. Nestor performs shape recognition by analyzing contour structures and generating projective-invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

Sketching leverages human skills for various purposes. In-Place Augmented Reality Sketching experiences build on the intuitiveness and flexibility of hand sketching for tasks like content creation. In this paper we explore the design space of In-Place Augmented Reality Sketching, with particular attention to content authoring in games. We propose a contextual model that offers a framework for the exploration of this design space by the research community. We describe a sketch-based AR racing game we developed to demonstrate the proposed model. The game is developed on top of our shape recognition and 3D registration library for mobile AR.

We present a framework for authoring three-dimensional virtual scenes for Augmented Reality (AR) which is based on hand sketching. Sketches consisting of multiple components are used to construct a 3D virtual scene augmented on top of the real drawing. Model structure and properties can be modified by editing the sketch itself and printed content can be combined with hand sketches to form a single scene. Authoring by sketching opens up new forms of interaction that have not been previously explored in Augmented Reality. To demonstrate the technology, we implemented an application that constructs 3D AR scenes of mechanical systems from freehand sketches, and animates the scenes using a physics engine. We provide examples of scenes composed from trihedral solid models, forces, and springs. Finally, we describe how sketch interaction can be used to author complicated physics experiments in a natural way.

In this paper we present Nestor, a system for real-time recognition and camera pose estimation from planar shapes. The system allows shapes that carry contextual meanings for humans to be used as augmented reality (AR) tracking fiducials. The user can teach the system new shapes at runtime by showing them to the camera. The learned shapes are then maintained by the system in a shape library. Nestor performs shape recognition by analyzing contour structures and generating projective invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

We introduce a novel touch-based interaction technique for tangible user interfaces (TUIs) in Augmented Reality (AR) applications. The technique allows for direct access and manipulation of virtual content on a registered tracking target, is robust and lightweight, and can be applied in numerous tracking and interaction scenarios.

The aim of our research is to create a system whereby human members of a team can collaborate in a natural way with robots. In this paper we describe a Wizard of Oz (WOZ) study conducted to find the natural speech and gestures people would use when interacting with a mobile robot as a team member. Results of the study show that in the beginning participants used simple speech, but once the users learned that the system understood more complicated speech, they began to use more spatially descriptive language. User responses indicate that gestures aided in spatial communication. The input mode that combined the use of speech and gestures was found to be best. We first discuss previous work and detail how our study contributes to this body of knowledge. Then we describe the design of our WOZ study and discuss the results and issues encountered during the completion of the experiment.

In this paper we evaluate the safety of the driver when using an embedded communication device while driving. As a part of our research, four different tasks were preformed with the device in order to evaluate the efficiency and safety of the drivers under three different conditions: one visual and two different auditory conditions. In the visual condition, various menu items were shown on a small LCD screen attached to the dashboard. In the auditory conditions, the same menu items were presented with spatial sounds distributed on a virtual ring around the user's head. The same custom-made interaction device attached to the steering wheel was used in all three conditions, enabling simple and safe interaction with the device while driving. The auditory interface proved to be as fast as the visual one, while at the same time enabling a significantly safer driving and higher satisfaction of the users. The measured workload also appeared to be lower when using the auditory interfaces.

In this paper we describe a Wizard of Oz (WOz) user study of an Augmented Reality (AR) interface that uses multimodal input (MMI) with natural hand interaction and speech commands. Our goal is to use a WOz study to help guide the creation of a multimodal AR interface which is most natural to the user. In this study we used three virtual object arranging tasks with two different display types (a head mounted display, and a desktop monitor) to see how users used multimodal commands, and how different AR display conditions affect those commands. The results provided valuable insights into how people naturally interact in a multimodal AR scene assembly task. For example, we discovered the optimal time frame for fusing speech and gesture commands into a single command. We also found that display type did not produce a significant difference in the type of commands used. Using these results, we present design recommendations for multimodal interaction in AR environments.

In this paper we present the results of our long term development of a mixed reality book. Most previous work in the area has focused on the technology of augmented reality books, such as providing registration using fiducial markers. In this work, however, we focused on exploring the design and development process of this type of application in a broader sense. We studied the semantics of a mixed reality book, the design space and the user experience with this type of interface.

In this paper we present a new vision-based approach for transmitting virtual models for augmented reality (AR). A two dimensional representation of the virtual models is embedded in a printed image. We apply image-processing techniques to interpret the printed image and extract the virtual models, which are then overlaid back on the printed image. The main advantages of our approach are: (1) the image of the embedded virtual models and their behaviors are understandable to a human without using an AR system, and (2) no database or network communication is required to retrieve the models. The latter is useful in scenarios with large numbers of users. We implemented an AR system that demonstrates the feasibility of our approach. Applications in education, advertisement, gaming, and other domains can benefit from our approach, since content providers need only to publish the printed content and all virtual information arrives with it.

We present a mobile phone based augmented reality (AR) assembly system that enable users to view complex models on their mobile phones. It is based on a client-server architecture, where complex model information is located on a PC, and a mobile phone with the camera is used as a thin client access device to this information. With this system users are able to see an AR view that provides step by step guidance for a real world assembly task. We also present results from a pilot user study evaluating the system, showing that people felt the interface was intuitive and very helpful in supporting the assembly task.

This paper introduces ComposAR, a tool to allow a wide audience to author AR and MR applications. It is unique in that it supports both visual programming and interpretive scripting, and an immediate mode for runtime testing. ComposAR is written in Python which means the user interface and runtime behavior can be easily customized and third-party modules can be incorporated into the authoring environment. We describe the design philosophy and the resulting user interface, lessons learned and directions for future research.

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.

In this paper we describe an exploration into the usability of spatial sound and multimodal interaction techniques for a mobile phone conferencing application. We compared traditional keypad based-interaction to that of a newer approach using the phone itself as a device to navigate within a virtual spatial auditory environment. While the traditional keypad interaction proved to be more straightforward to use, there was no significant impact on task completion times or number of interaction movements made between the techniques. Overall, users felt that the spatial audio application supported group awareness while aiding peripheral task monitoring. They also felt it aided the feeling of social connectedness and offered enhanced support for communication.

In this paper, we describe an Augmented Reality Art installation, which reacts to user behaviour using Multimodal analysis of affective signals. The installation features a virtual tree, whose growth is influenced by the perceived emotional response from spectators. The system implements a 'magic mirror' paradigm (using a large-screen display or projection system) and is based on the ARToolkit with extended representations for scene graphs. The system relies on a PAD dimensional model of affect to support the fusion of different affective modalities, while also supporting the representation of affective responses that relate to aesthetic impressions. The influence of affective input on the visual component is achieved by mapping affective data to an L-System governing virtual tree behaviour. We have performed an early evaluation of the system, both from the technical perspective and in terms of user experience. Post-hoc questionnaires were generally consistent with data from multimodal affective processing, and users rated the overall experience as positive and enjoyable, regardless of how proactive they were in their interaction with the installation.

In this paper we present a mobile phone based augmented reality (AR) assembly system that enables users to view complex models on their mobile phones. It is based on a client/server architecture, where complex model information is located on a PC, and a camera phone is used as a thin client access device to augment still images of the assembly site with animated AR sequences.

As computers become more invisible, Augmented Reality (overlaying virtual images on the real world) is an increasingly important application area for computer graphics and user interface design. This tutorial will provide a detailed introduction to AR interface design and research, with reviews of topics as such as tracking and registration, interaction techniques, design principles, and usability evaluation, as well as areas for future AR research. Portions of the tutorial will be devoted to the ARToolKit and osgART open-source software tools for building AR applications. Participants will leave knowing how to build AR applications that provide a high quality user experience.

VoodooSketch is a system that extends interactive surfaces with physical interface palettes on which users can dynamically deploy controls as shortcut to application functionality. The system provides physical 'plug and play' controls as well as support for sketching of controls, and allows controls to be associated with application functions via handwritten labels. The system uses a special digital pen, which writes 'real' ink on the palettes while functioning as a digital input device on the interactive surfaces. The palettes can be seamlessly integrated into existing applications, be appropriated by the user to suit different input requirements, and support new interaction styles across multiple surfaces, palettes and users.

Recently researchers have shown that it is possible to use GPU hardware for image processing and computer vision algorithms. We have been exploring how to use GPU hardware to improve marker- based tracking for AR Applications. In this paper we describe our findings and explored issues in the context of a standard fiducial tracking pipeline. We demonstrate the implementation of a template matching process on the GPU and the performance improvement gained in comparison to a traditional CPU implementation.

In this paper we present the process of designing and developing a fully functional and interactive AR handheld device. Based on a human-centered approach we describe the results of our exploration; from the contextual usage to the different rapid prototyped models and the evaluation activities. We also discuss some recommendations for how to use similar methods for developing future new AR physical interfaces.

The Magic Lens concept is a focus and context technique which facilitates the visualization of complex and dense data. In this paper, we propose a new type of 3D tangible Magic Lens in the form of a flexible sheet. We describe new interaction techniques associated with this tool, and demonstrate how it can be applied in different AR applications.

In this paper, we explore ways to combine the video of a remote person with a shared tabletop display to best emulate face-to-face collaboration. Using a simple photo application we compare a variety of social and performance measures of collaboration of a standard non-spatial 2D interface with two approaches for adding spatial cues to videoconferencing: one based on simulated immersive 3D, the other based on video streams in a physically fixed arrangement around an interactive table. A face-to-face condition is included as a 'gold-standard' control. As expected, social presence and task measures were superior in the face-to-face condition, but there were also important differences between the 2D and spatial interfaces. In particular, the spatial interfaces positively influenced social presence and copresence measures in comparison to 2D, but the task measures favored the two-dimensional interface.

Transitional Interfaces have emerged as a new way to interact and collaborate between different interactive spaces such as reality, virtual reality and augmented reality environments. In this paper we explore this concept further. We introduce a descriptive model of the concept, its collaborative aspect and how it can be generalized to describe natural and continuous transitions between contexts (e.g. across space, scale, viewpoints, and representation).

This paper describes an augmented reality (AR) multimodal interface that uses speech and paddle gestures for interaction. The application allows users to intuitively arrange virtual furniture in a virtual room using a combination of speech and gestures from a real paddle. Unlike other multimodal AR applications, the multimodal fusion is based on the combination of time-based and semantic techniques to disambiguate a users speech and gesture input. We describe our AR multimodal interface architecture and discuss how the multimodal inputs are semantically integrated into a single interpretation by considering the input time stamps, the object properties, and the user context.

Hand-held devices possess a large potential as an interaction device for their today's ubiquity, and present us with an opportunity to devise new and unique ways of interaction as a smart device with multi-modal sensing and display capabilities. This paper introduces user interaction techniques (for selection, translation, scaling and rotation of objects) using a camera-equipped hand-held device such as a mobile phone or a PDA for large shared environments. We propose three intuitive interaction techniques for 2D and 3D objects in such an environment. The first approach uses the motion flow information to estimate the relative motion of the hand-held device and interact with the large display. The marker-object and marker-cursor approaches both use software markers on the interaction object or on the cursor for the various interactive tasks. The proposed interaction techniques can be further combined with many auxiliary functions and wireless services (of the hand-held devices) for seamless information sharing and exchange among multiple users. A formal usability analysis is currently on-going.

Recently augmented reality (AR) technology has been used to develop the next generation collaborative interfaces. First results have shown the value of using AR for co-located tasks based on egocentric viewpoints. In contrast, virtual reality (VR) seems to offer interesting advantages for immersive collaborative experiences with egocentric viewpoints. We focus on a new area: a mixed collaboration between AR and VR environments. We present a new conceptual model of transitional interfaces that allow users to move between AR and VR viewpoints. We then describe the results of a quantitative evaluation with an AR exocentric viewpoint and a VR egocentric viewpoint for a navigational task. We also conducted a second experiment on the impact of the relationship between the interaction and visualization space in mixed collaboration. Results of these studies can provide a better understanding of how to design interfaces for multispace and transitional collaboration.

We describe the AR Mask, a novel handheld augmented reality (AR) input and display device that not only provides support for a full range of traditional interaction techniques, but also facilitates new metaphors. Unlike other AR display and input technologies, our device consolidates input and output within a single piece of hardware. We demonstrate how this design provides a closed control loop between the user's input modalities and natural sensory receptors.

Mobile phones are an ideal platform for augmented reality. We describe how they also can be used to support face to face collaborative AR applications. We have created a custom port of the ARToolKit library to the Symbian mobile phone operating system and then developed a sample collaborative AR game based on this. We describe the game in detail and user feedback from people who have played it. We also provide general design guidelines that could be useful for others who are developing mobile phone collaborative AR applications.

This paper presents the findings of a comparative study investigating different input interfaces for a mobile phone games application. A standard mobile phone joystick interface is compared with a phone camera interface to detect the phone translation and tilt to control a ball's movement within various levels of difficulty of a virtual maze game. Results indicate that the joystick control provided the fastest completion times for each game, but with the lowest levels of user engagement. The Tilt interface, although perceived as challenging by the participants, provided the greatest level of user involvement, independent of game complexity. The design of appropriate human interfaces which go beyond the standard phone keypad is suggested.

This paper presents the findings of a comparative study investigating different input interfaces for a mobile phone games application. A standard mobile phone joystick interface is compared with a phone camera interface to detect the phone translation and tilt to control a ball's movement within various levels of difficulty of a virtual maze game. Game completion times together with the resultant user experience for each of the games was collected during the studies. Results indicate that the joystick control provided the fastest completion times for each game, but with the lowest levels of user engagement. The Tilt interface, although perceived as challenging by the participants, provided the greatest level of user involvement, independent of game complexity. The design of appropriate human interfaces which go beyond the standard phone keypad is suggested. The selection and design of these interfaces should also consider the intended user experience for the game.

In this paper we describe a mobile phone based Augmented Reality application for 3D scene assembly. Augmented Reality on mobile phones extends the interaction capabilities on such handheld devices. It adds a 6 DOF isomorphic interaction technique for manipulating 3D content. We give details of an application that we believe to be the first where 3D content can be manipulated using both the movement of a camera tracked mobile phone and a traditional button interface as input for transformations. By centering the scene in a tangible marker space in front of the phone we provide a mean for bimanual interaction. We describe the implementation, the interaction techniques we have developed and initial user response to trying the application.

We describe an augmented reality system for superimposing three-dimensional (3-D) live content onto two-dimensional fiducial markers in the scene. In each frame, the Euclidean transformation between the marker and the camera is estimated. The equivalent virtual view of the live model is then generated and rendered into the scene at interactive speeds. The 3-D structure of the model is calculated using a fast shape-from-silhouette algorithm based on the outputs of 15 cameras surrounding the subject. The novel view is generated by projecting rays through each pixel of the desired image and intersecting them with the 3-D structure. Pixel color is estimated by taking a weighted sum of the colors of the projections of this 3-D point in nearby real camera images. Using this system, we capture live human models and present them via the augmented reality interface at a remote location. We can generate 384/spl times/288 pixel images of the models at 25 fps, with a latency of <100 ms. The result gives the strong impression that the model is a real 3-D part of the scene.

In this paper, we suggest a new approach for authoring tangible augmented reality applications, called 'immersive authoring.' The approach allows the user to carry out the authoring tasks within the AR application being built, so that the development and testing of the application can be done concurrently throughout the development process. We describe the functionalities and the interaction design for the proposed authoring system that are specifically targeted for intuitive specification of scenes and various object behaviors. Several cases of applications developed using the authoring system are presented. A small pilot user study was conducted to compare the proposed method to a non-immersive approach, and the results have shown that the users generally found it easier and faster to carry out authoring tasks in the immersive environment.

Provides an abstract of the tutorial presentation and a brief professional biography of the presenter. The complete presentation was not made available for publication as part of the conference proceedings.

In this paper we explore the effect of using augmented reality (AR) for three-dimensional graph link analysis. Two experiments were conducted. The first was designed to compare a tangible AR interface to a desktop-based interface. Different modes of viewing network graphs were presented using a variety of interfaces. The results of the first experiment show that a tangible AR interface is well suited to link analysis. The second experiment was designed to test the effect of stereographic viewing on graph comprehension. The results show that stereographic viewing has little effect on comprehension and performance. These experiments add support to the work of Ware and Frank, whose studies showed that depth and motion cues provide huge gains in spatial comprehension and accuracy in link analysis.

An ideal augmented reality (AR) display for multi-user co-located collaboration should have following three features: 1) any virtual object should be able to be shown at any arbitrary position, e.g. a user can see a virtual object in front of other users' faces. 2) Correct occlusion of virtual and real objects should be supported. 3) The real world should be naturally and clearly visible, which is important for face-to-face conversation. We have been developing an optical see-through display, ELMO (Enhanced see-through display using an LCD panel for Mutual Occlusion), that satisfies these three requirements. While previous prototype systems were not practical due to their size and weight, we have come up with an improved optics design which has reduced size and is lightweight enough to wear. In this paper, the characteristics of typical multi-user three-dimensional displays are summarized and the design details of the latest optics are then described. Finally, a collaborative AR application employing the new display and its user experience are explained.

Current augmented reality (AR) applications require that the application software be written to support a specific AR interface set up. WireAR was developed to enable output from any OpenGL application to be viewed in an AR fashion. This enables the output from any legacy graphical or scientific visualization applications to be viewed in a collaborative AR setting. This demonstration shows how the output of standard desktop visualization programs can be embedded into an augmented reality experience.

The authors present an analysis of communication behavior in face-to-face collaboration using a multi-user augmented reality (AR) interface. 2 experiments were conducted. In the 1st experiment, collaboration with AR technology was compared with more traditional unmediated and screen-based collaboration. In the 2nd experiment, the authors compared collaboration with 3 different AR displays. Several measures were used to analyze communication behavior, and the authors found that users exhibited many of the same behaviors in a collaborative AR interface as in face-to-face unmediated collaboration. However, user communication behavior changed with the type of AR display used. The authors describe implications of these results for the design of collaborative AR interfaces and directions for future research.

We describe a real-time 3-D augmented reality video- conferencing system. With this technology, an observer sees the real world from his viewpoint, but modified so that the image of a remote collaborator is rendered into the scene. We register the image of the collaborator with the world by estimating the 3-D transformation between the camera and a fiducial marker. We describe a novel shape- from-silhouette algorithm, which generates the appropriate view of the collaborator and the associated depth map at 30 fps. When this view is superimposed upon the real world, it gives the strong impression that the collaborator is a real part of the scene. We also demonstrate interaction in virtual environments with a live fully 3-D collaborator. Finally, we consider interaction between users in the real world and collaborators in a virtual space, using a tangible AR interface.

This paper presents an interactive theatre based on an embodied mixed reality space and wearable computers. Embodied computing mixed reality spaces integrate ubiquitous computing, tangible interaction and social computing within a mixed reality space, which enables intuitive interaction with physical world and virtual world. We believe it has potential advantages to support novel interactive theatre experiences. Therefore, we explored the novel interactive theatre experience supported in the embodied mixed reality space, and implemented live 3D characters to interact with user in such a system.

This paper describes an augmented reality system that incorporates a real-time dense stereo vision system. Analysis of range and intensity data is used to perform two functions: 1) 3D detection and tracking of the user's fingertip or a pen to provide natural 3D pointing gestures, and 2) computation of the 3D position and orientation of the user's viewpoint without the need for fiducial mark calibration procedures, or manual initialization. The paper describes the stereo depth camera, the algorithms developed for pointer tracking and camera pose tracking, and demonstrates their use within an application in the field of oil and gas exploration.

We conducted two experiments comparing communication behaviors of co-located users in collaborative augmented reality (AR) interfaces. In the first experiment, we compared optical, stereo- and mono-video, and immersive head mounted displays (HMDs) using a target identification task. It was found that differences in the real world visibility severely affect communication behaviors. The optical see-through case produced the best results with the least extra communication needed. Generally, the more difficult it was to use non-verbal communication cues, the more people resorted to speech cues to compensate. In the second experiment, we compared three different combinations of task and communication spaces using a 2D icon design task with optical see-through HMDs. It was found that the spatial relationship between the task and communication spaces also severely affected communication behaviors. Placing the task space between the subjects produced the most active behaviors in terms of initiatory body languages and utterances with least miscommunications.

We present a complete system for live capture of 3D content and simultaneous presentation in augmented reality. The user sees the real world from his viewpoint, but modified so that the image of a remote collaborator is rendered into the scene. Fifteen cameras surround the collaborator, and the resulting video streams are used to construct a three-dimensional model of the subject using a shape-from-silhouette algorithm. Users view a two-dimensional fiducial marker using a video-see-through augmented reality interface. The geometric relationship between the marker and head-mounted camera is calculated, and the equivalent view of the subject is computed and drawn into the scene. Our system can generate 384 /spl times/ 288 pixel images of the models at 25 fps, with a latency of < 100 ms. The result gives the strong impression that the subject is a real part of the 3D scene. We demonstrate applications of this system in 3D videoconferencing and entertainment.

The research community is challenged to investigate the factors that make virtual reality technologies effective and productive. Realising the goals of virtual reality systems and harnessing them to successful applications can be accomplished only by employing robust metrics and formal experimentation. Subsequently, the quality of interfaces and platforms implemented could be assessed.

In its simplest form, reality is merely information that is presented or acquired. Mixed Reality (MR) is built around the integration of real world physical and computer generated virtual information. We do not use the term augmented reality (AR) because we view the merging of both worlds as a symbiosis, with desirable properties from each accentuated and complementing each other, rather than the enhancement of one with the other. Collaborative MR allows multiple participants to simultaneously share a physical space while being surrounded by a virtual space that is registered with the physical. Because the MR world inherits the properties of real and virtual worlds, it is rich with social context, spatial cues, and tangible objects from the real world as well as flexible digital information from virtual. We believe that Mixed Reality is a medium, largely unexplored, but very well suited for face-to-face collaboration.

In this paper, we argue for embodied corrversational characters as the logical extension of the metaphor of human - computer interaction as a conversation. We argue that the only way to fully model the richness of human I&+ to-face communication is to rely on conversational analysis that describes sets of conversational behaviors as fi~lfilling conversational functions, both interactional and propositional. We demonstrate how to implement this approach in Rea, an embodied conversational agent that is capable of both multimodal input understanding and output generation in a limited application domain. Rea supports both social and task-oriented dialogue. We discuss issues that need to be addressed in creating embodied conversational agents, and describe the architecture of the Rea interface.

The acceptance of virtual environment (VE) technology requires scrupulous optimization of the most basic interactions in order to maximize user performance and provide efficient and enjoyable virtual interfaces. Motivated by insufficient understanding of the human factors design implications of interaction techniques and tools for virtual interfaces, this paper presents results of a formal study that compared two basic interaction metaphors for egocentric direct manipulation in VEs, virtual hand and virtual pointer, in object selection and positioning experiments. The goals of the study were to explore immersive direct manipulation interfaces, compare performance characteristics of interaction techniques based on the metaphors of interest, understand their relative strengths and weaknesses, and derive design guidelines for practical development of VE applications.

No abstract available.

Developing virtual reality (VR) applications which enable actual work over a period of time requires optimization of the most basic interactions, such as object manipulation, so that the immersed participant can concentrate on higher-level tasks rather than on lowlevel motor activities. This paper presents a framework and experimental testbed for studies of VR object manipulation techniques. The framework provides a systematic task analysis of immersive manipulation and suggests a user-specific non-Euclidean system for the measurement of VR spatial relationships. The Virtual Reality Manipulation Assessment Testbed (VRMAT) is a practical implementation of the framework and is a flexible tool allowing in-depth experimental studies of immersive manipulation. Pilot studies have been conducted to evaluate this framework and testbed and to establish a baseline for further development. Keywords: immersive virtual reality, VR user interfaces, VR manipulation techniques, user studies, experimen...

The Go-Go immersive interaction technique uses the metaphor of interactively growing the user’s arm and non-linear mapping for reaching and manipulating distant objects. Unlike others, our technique allows for seamless direct manipulation of both nearby objects and those at a distance.