Cross-functional teams struggle when static collaboration tools fail to keep pace with dynamic conversations. Through a formative study with seven professionals, we identified a critical gap: designers and developers speak different vocabularies, causing semantic misalignments. We present Cognitive Bridge, an AI system that monitors multimodal cues (facial expressions, speech, workspace activity) to detect emerging misunderstandings, then generates adaptive boundary objects, visual diagrams, wireframes, and flowcharts that translate between professional perspectives in real-time. Our controlled study with 16 designer-developer dyads found that Cognitive Bridge reduced communication conflicts by 47% and increased implementable solutions by 34% compared to baseline tools. However, analysis revealed a solution-exploration tradeoff: while AI accelerated alignment, it risked premature convergence that constrained creative exploration. We contribute: (1) a novel system for AI-generated boundary objects, and (2) design implications for balancing cognitive scaffolding with creative agency preservation.

Integrating non-verbal cues into Mixed Reality Agents (MiRAs) enhances their ability to engage users and foster socially rich interactions. This paper investigates the role of locomotion and body posture in shaping user engagement, social presence, and interaction quality through two user studies involving a turn-based Gobang game. From these studies we found that in a competitive context, MiRAs’ locomotion and posture enhanced social presence and engagement, but while in a cooperative context, these behaviors fostered rapport but not trust. By integrating subjective, behavioral, and physiological measures, including EEG, this study provides a holistic understanding of MiRAs’ impact. The findings offer actionable design implications for creating engaging and socially effective virtual agents, advancing the field of human-agent interaction in Mixed Reality. Future research directions include exploring long-term effects, using diverse application domains, and supporting multimodal interactions.

Video conferencing is essential for remote collaboration, but it often leads to fatigue, reduced social presence and ineffective communication. Traditional human facilitators can address these challenges but cannot scale to meet the demands of countless daily virtual meetings across organisations. To address these challenges, we introduce Cognitive Load and Affect Aware Agent (CLARA), an AI-mediated facilitator that enhances group decision-making by dynamically managing cognitive load and affective engagement. CLARA employs real-time multimodal assessment of group states, providing adaptive cognitive prompts to optimise task focus and affective cues to foster positive dynamics. In a controlled study (N = 48), we compared Baseline, Cognitive Feedback (CF), Affective Feedback (AF) and Combined Feedback (CAF) conditions. Results show CAF significantly improved task performance, reduced mental demand and enhanced social presence, outperforming all other conditions. Participants rated CAF as having the highest level of facilitator expertise and preference. These findings highlight the benefits of integrated AI-driven facilitation, offering design insights for human-centred, effective virtual collaboration tools that balance task efficiency with positive socio-emotional engagement.

Mixed Reality Agents (MiRAs) have been extensively studied to enhance virtual-physical interactions, using their ability to exist in both virtual and physical environments. However, little research has focused on enhancing perceived empathy in MiRAs, despite its potential for agent-assisted therapy, education, and training. To fill this gap, we investigate the impact of an Empathic Mixed Reality agent (EMiRA) that adapts to users' physiological states and physical events in a shooting game. We found that this adaptation enhanced users' social perceptions of the agent, including social presence, social connectedness, and perceived empathy. Physiological adaptation increased paternalism and reduced user dominance, while physical adaptation had no such effect. We discuss these findings and provide design implications for future EMiRAs.

Non-verbal cues like locomotion and posture influence users’ perceptions of Mixed Reality Agents (MiRAs). While Electroencephalography (EEG) captures cognitive responses, the influence of MiRAs’ locomotion and postures on brain activity remains underexplored. Additionally, few studies integrate subjective and behavioral measures with EEG to evaluate these cues’ impact on social perception. To address this, we conducted a within-subject study where participants played Gobang against three virtual agents in mixed reality: 1) a speech-only agent (S), 2) an embodied agent with speech and locomotion (S + L), and 3) an embodied agent with speech, locomotion, and posture (S + L + P). Results showed the S + L + P agent had higher engagement measured by the questionnaire but a lower EEG-based engagement index at AF3 than the S + L agent. Besides, the S + L + P was also rated higher in social presence, engagement, and emotional arousal than the S condition; No behavioral differences were observed. We discuss how MiRAs’ locomotion and posture affect users’ social perception and provide design implications for future human-agent interactions.

Understanding the relationship between cognitive load and affective state in remote work is vital for designing intuitive collaboration. We present CoAffinity, a multimodal dataset encompassing eight structured remote-work tasks, during which 39 participants provided self-reported measures (arousal, valence, positive/negative affect, and cognitive-load) while being recorded via audio, video, and physiological signals (PPG and GSR). Spanning over 38 hours of annotated data, our approach involved precise timestamp alignment, short and long-session labelling, and subsequent machine-learning and deep-learning benchmarks. Key findings show that integrating multiple modalities, especially physiological data, significantly improves the detection of cognitive load and emotion, while group synchrony metrics highlight how physiological coherence shifts under varied task demands. By capturing complex cognitive-emotional dynamics in realistic remote settings, CoAffinity aims to advance affective computing, inform human-computer interaction research, and foster more empathetic remote collaboration tools

In human-agent interaction, establishing trust and a social bond with the agent is crucial to improving communication quality and performance in collaborative tasks. This paper investigates how a Mixed Reality Agent’s (MiRA) ability to acknowledge a user’s physiological state affects perceptions such as empathy, social connectedness, presence, and trust. In a within-subject study with 24 subjects, we varied the companion agent’s awareness during a mixed-reality first-person shooting game. Three agents provided feedback based on the users’ physiological states: (1) No Awareness Agent (NAA), which did not acknowledge the user’s physiological state; (2) Random Awareness Agent (RAA), offering feedback with varying accuracy; and (3) Accurate Awareness Agent (AAA), which provided consistently accurate feedback. Subjects reported higher scores on perceived empathy, social connectedness, presence, and trust with AAA compared to RAA and NAA. Interestingly, despite exceeding NAA in perception scores, RAA was the least favored as a companion. The findings and implications for the design of MiRA interfaces are discussed, along with the limitations of the study and directions for future work.

In this research, we investigate the effectiveness of asymmetric interactions (HandStylus, HandController, and TwoHands) in Augmented Reality (AR), Virtual Reality (VR), and Extended Reality (XR) for 3D digital drawing overlaying on physical and virtual objects. We evaluate the input accuracy and fatigue of these object-based 3D drawing experiences using quantitative measurements and further explore the correlation between these outcomes with subjective questionnaires. We found significant independence between environments and interface types, which considerably influence the performance and usability of 3D immersive drawing. We noted discrepancies between users’ subjective experiences and objective performance. Specifically, although AR drawing on physical objects provides superior accuracy and minimal muscle fatigue due to tangible feedback, and the TwoHands interaction offers the highest precision, the subjective results show the reverse outcome. Based on these findings, we propose design recommendations and discuss directions for future research in immersive drawing environments.

In collaborative settings where multiple individuals are tasked with completing a shared goal, understanding one’s partner’s emotional state could be crucial for achieving a successful outcome. This is particularly relevant in remote collaboration contexts, where physical distance can impede understanding, empathy, and mutual comprehension between partners. In this paper, we demonstrate representing emotional patterns from physiological data in a shared Virtual Reality (VR) environment, and explore how it impacted communication styles. A user study investigated the potential effects of this emotional representation in fostering empathetic communication during remote collaboration. The study’s findings revealed that although there was minimal variance in the workload associated with observing physiological cues, participants generally preferred monitoring their partner’s attentional state. However, with the assembly task chosen, most participants only directed a minimal proportion of their attention toward the physiological cues displayed by their partner, and were frequently uncertain of how to interpret and use the information obtained. We also discuss limitations of the research and opportunities for future work.

This research investigates fast and precise Virtual Reality (VR) sketching methods with different tool-based asymmetric interfaces. In traditional real-world drawing, artists commonly employ an asymmetric interaction system where each hand holds different tools, facilitating diverse and nuanced artistic expressions. However, in virtual reality (VR), users are typically limited to using identical tools in both hands for drawing. To bridge this gap, we aim to introduce specifically designed tools in VR that replicate the varied tool configurations found in the real world. Hence, we developed a VR sketching system supporting three hybrid input techniques using a standard VR controller, a VR stylus, or a data glove. We conducted a formal user study consisting of an internal comparative experiment with four conditions and three tasks to compare three asymmetric input methods with each other and with a traditional symmetric controller-based solution based on questionnaires and performance evaluations. The results showed that in contrast to symmetric dual VR controller interfaces, the asymmetric input with gestures significantly reduced task completion times while maintaining good usability and input accuracy with a low task workload. This shows the value of asymmetric input methods for VR sketching. We also found that the overall user experience could be further improved by optimizing the tracking stability of the data glove and the VR stylus.

Many Virtual Reality (VR) experiences, such as learning tools, would benefit from utilising mental states such as cognitive load. Increases in cognitive load (CL) are often reflected in the alteration of physiological responses, such as pupil dilation (PD), electrodermal cctivity (EDA), heart rate (HR), and electroencephalography (EEG). However, the relationship between these physiological responses and cognitive load are usually measured while participants sit in front of a computer screen, whereas VR environments often require a high degree of physical movement. This physical activity can affect the measured signals, making it unclear how suitable these measures are for use in interactive Virtual Reality (VR).

Physics-driven predictions of soft tissue mechanics are vital for various medical interventions. Insights on the mechanical properties of soft tissues are essential for obtaining personalised predictions from these models. This study aims to provide a workflow to identify the material parameters of soft homogeneous materials under gravity loading using 3D surface geometrical measurements acquired from a wearable augmented reality (AR) headset’s depth camera. Preliminary results show that the parameter estimation procedure can successfully recover the ground truth material parameter C1 of a cantilever beam using synthetic surface data. This workflow could be used for real-time navigational guidance during soft tissue treatment procedures.

Consistent visual and haptic feedback is an important way to improve the user experience when interacting with virtual objects. However, the perception provided in Augmented Reality (AR) mainly comes from visual cues and amorphous tactile feedback. This work explores how to simulate positional vibrotactile feedback (PVF) with multiple vibration motors when colliding with virtual objects in AR. By attaching spatially distributed vibration motors on a physical haptic proxy, users can obtain an augmented collision experience with positional vibration sensations from the contact point with virtual objects. We first developed a prototype system and conducted a user study to optimize the design parameters. Then we investigated the effect of PVF on user performance and experience in a virtual and real object alignment task in the AR environment. We found that this approach could significantly reduce the alignment offset between virtual and physical objects with tolerable task completion time increments. With the PVF cue, participants obtained a more comprehensive perception of the offset direction, more useful information, and a more authentic AR experience.

In this paper, we explore how virtual replicas can enhance Mixed Reality (MR) remote collaboration with a 3D reconstruction of the task space. People in different locations may need to work together remotely on complicated tasks. For example, a local user could follow a remote expert's instructions to complete a physical task. However, it could be challenging for the local user to fully understand the remote expert's intentions without effective spatial referencing and action demonstration. In this research, we investigate how virtual replicas can work as a spatial communication cue to improve MR remote collaboration. This approach segments the foreground manipulable objects in the local environment and creates corresponding virtual replicas of physical task objects. The remote user can then manipulate these virtual replicas to explain the task and guide their partner. This enables the local user to rapidly and accurately understand the remote expert's intentions and instructions. Our user study with an object assembly task found that using virtual replica manipulation was more efficient than using 3D annotation drawing in an MR remote collaboration scenario. We report and discuss the findings and limitations of our system and study, and present directions for future research.

There are decades of work investigating how to support users in inhabiting avatars’ bodies in immersive experiences, however we are still learning about how Virtual Reality (VR) can support people in more deeply inhabiting their own bodies. Over the course of five action research workshops, we explored the design space for how VR can support feeling embodied and generating movement within a dance context. We explored design factors and participants’ experiences within: games intended to stimulate physical movement, single player and multiplayer 3D painting, 360 live video, and custom real-time audio and visual feedback based on motion capture. We found that participants spontaneously explored collaboration including synchronous, asynchronous, remote and collocated. Where there were mismatches between visual and kinesthetic perceptions, participants explored how to recalibrate, alternate between, and integrate perceptions. Participants were compelled to explore their control of real-time effects, which led to rich movement generation as we iterated through the effects’ parameters. We present a design space that encapsulates the insights of our action research, with axes for control, collaboration, auditory and visual feedback. We discuss implications for the support of immersive embodiment in dance.

We present a Virtual Reality (VR) game, PlayMeBack, to investigate cognitive load measurement in interactive VR environments using pupil dilation, Galvanic Skin Response (GSR), Electroencephalogram (EEG) and Heart Rate (HR). The user is shown different patterns of tiles lighting up and is asked to replay the pattern back pressing the tiles in the same sequence they lit up. The task difficulty depends on the length of the observed pattern (3-6 keys). This task is designed to explore the effect of cognitive load on physiological cues, and if pupil dilation, EEG, GSR and HR can be used as measures of cognitive load.

Many studies have focused on Virtual Reality (VR) frameworks for remotely controlling robotic systems. Although VR systems have been used to teleoperate robots in simple scenarios, their effectiveness in terms of accuracy, speed, and usability has not been rigorously evaluated for complex tasks that require accurate trajectories. In this work, an Enhanced Virtual Reality (EVR) framework for robotic teleoperation is evaluated to assess if it can be efficiently used in complex tasks that require accurate control of the robotic end-effector. The environment and the employed robot are captured using RGB-D cameras, while the remote user controls the motion of the robot with VR controllers. The captured data are transmitted and reconstructed in 3D so as to allow the remote user to monitor the task execution progress in real time, using a VR headset. The EVR system is compared with two other interface alternatives: i) teleoperation in pure VR (the model of the robot is rendered with respect to its real joint states), and ii) teleoperation in EVRR (the model of the robot is superimposed on the real robot). The results show that pure point cloud interfaces suffer from visualization issues, reducing the effectiveness of the robot teleoperation. However, the accuracy and user experience can be greatly improved by including the robot model.

Supporting natural communication cues is critical for people to work together remotely and face-to-face. In this paper we present a Mixed Reality (MR) remote collaboration system that enables a local worker to share a live 3D panorama of his/her surroundings with a remote expert. The remote expert can also share task instructions back to the local worker using visual cues in addition to verbal communication. We conducted a user study to investigate how sharing augmented gaze and gesture cues from the remote expert to the local worker could affect the overall collaboration performance and user experience. We found that by combing gaze and gesture cues, our remote collaboration system could provide a significantly stronger sense of co-presence for both the local and remote users than using the gaze cue alone. The combined cues were also rated significantly higher than the gaze in terms of ease of conveying spatial actions.

In this work, a Mixed Reality (MR) system is evaluated to assess whether it can be efficiently used in teleoperation tasks that require an accurate control of the robot end-effector. The robot and its local environment are captured using multiple RGB-D cameras, and a remote user controls the robot arm motion through Virtual Reality (VR) controllers. The captured data is streamed through the network and reconstructed in 3D, allowing the remote user to monitor the state of execution in real time through a VR headset. We compared our method with two other interfaces: i) teleoperation in pure VR, with the robot model rendered with the real joint states, and ii) teleoperation in MR, with the rendered model of the robot superimposed on the actual point cloud data. Preliminary results indicate that the virtual robot visualization is better than the pure point cloud for accurate teleoperation of a robot arm.

Conventional 2D touch-based interaction methods for handheld Augmented Reality (AR) cannot provide intuitive 3D interaction due to a lack of natural gesture input with real-time depth information. The goal of this research is to develop a natural interaction technique for manipulating virtual objects in 3D space on handheld AR devices. We present a novel method that is based on identifying the positions and movements of the user's fingertips, and mapping these gestures onto corresponding manipulations of the virtual objects in the AR scene. We conducted a user study to evaluate this method by comparing it with a common touch-based interface under different AR scenarios. The results indicate that although our method takes longer time, it is more natural and enjoyable to use.