Creating interior layout designs has numerous applications, including virtual reality, architectural visualization and real estate planning. Generating realistic and functional indoor scenes requires a nuanced understanding of spatial configurations and human-centered design principles. We propose InteriorAgent, an LLM-agent-driven framework for text-to-3D indoor scene generation that produces scenes with visual quality and functional utility that significantly surpass prior works. We achieve this through several key advantages of InteriorAgent: (1) encoding of interior design principles with a novel scene description language, (2) aesthetics and functionality through synthesis tools that satisfy design principles, (3) realism and prompt adherence with optimization tools that ensure ergonomics and iterative constraint satisfaction, (4) extensibility with a framework that allows incorporating even mature, complex tools like diffusion models, LLMs and 3D generation repositories. We evaluate InteriorAgent through a user study, where participants strongly favor its generated scenes over prior state-of-the-art methods. Additionally, we demonstrate novel applications uniquely enabled by InteriorAgent, including language-based scene editing and seamless tool integration for new tasks. Code and data will be publicly released.

Cross-functional teams struggle when static collaboration tools fail to keep pace with dynamic conversations. Through a formative study with seven professionals, we identified a critical gap: designers and developers speak different vocabularies, causing semantic misalignments. We present Cognitive Bridge, an AI system that monitors multimodal cues (facial expressions, speech, workspace activity) to detect emerging misunderstandings, then generates adaptive boundary objects, visual diagrams, wireframes, and flowcharts that translate between professional perspectives in real-time. Our controlled study with 16 designer-developer dyads found that Cognitive Bridge reduced communication conflicts by 47% and increased implementable solutions by 34% compared to baseline tools. However, analysis revealed a solution-exploration tradeoff: while AI accelerated alignment, it risked premature convergence that constrained creative exploration. We contribute: (1) a novel system for AI-generated boundary objects, and (2) design implications for balancing cognitive scaffolding with creative agency preservation.

The Wizard of Oz (WoZ) method is a common and popular approach for simulating interactive systems in Human-Computer Interaction (HCI). Running such studies is demanding for researchers because the human wizard must manage human–agent interactions in real time while keeping participants safe and the interaction natural. Many WoZ systems struggle to reproduce complex agent behaviours without minimal delays or heavy workload for the moderator. We introduce AI of Oz, a framework that uses large language models to support researchers by monitoring ongoing interactions, detecting sensitive moments, and suggesting contextually appropriate responses. In a study with 20 HCI-related researchers, the system improved participants’ ability to manage interactions and maintain control compared to a version without AI support. We outline implications for WoZ research and note current limitations and future directions.

The emergence of generative artificial intelligence (GenAI), combined with immersive virtual reality (VR), enables the rapid creation of personalised virtual content from simple text prompts, holding potential for emotional support. However, most current VR systems rely on pre-authored content and limit user agency in designing emotionally meaningful experiences. We introduce OasisMind, an AI-assisted VR system that empowers users to co-create 360° environments, corresponding ambient soundscapes, and context-aware digital companions through natural language prompts. In a user study (N=24), we observed how participants constructed virtual worlds for emotionally meaningful use cases and compared their creations to validated, pre-defined VR scenes recommended by previous research. Our results indicate a subjective preference for self-created environments, while no significant differences were observed in perceived satisfaction or presence between conditions. These findings suggest that user agency contributes to the emotional resonance of virtual experiences and inform the design of future personalised companion systems.

Integrating non-verbal cues into Mixed Reality Agents (MiRAs) enhances their ability to engage users and foster socially rich interactions. This paper investigates the role of locomotion and body posture in shaping user engagement, social presence, and interaction quality through two user studies involving a turn-based Gobang game. From these studies we found that in a competitive context, MiRAs’ locomotion and posture enhanced social presence and engagement, but while in a cooperative context, these behaviors fostered rapport but not trust. By integrating subjective, behavioral, and physiological measures, including EEG, this study provides a holistic understanding of MiRAs’ impact. The findings offer actionable design implications for creating engaging and socially effective virtual agents, advancing the field of human-agent interaction in Mixed Reality. Future research directions include exploring long-term effects, using diverse application domains, and supporting multimodal interactions.

Video conferencing is essential for remote collaboration, but it often leads to fatigue, reduced social presence and ineffective communication. Traditional human facilitators can address these challenges but cannot scale to meet the demands of countless daily virtual meetings across organisations. To address these challenges, we introduce Cognitive Load and Affect Aware Agent (CLARA), an AI-mediated facilitator that enhances group decision-making by dynamically managing cognitive load and affective engagement. CLARA employs real-time multimodal assessment of group states, providing adaptive cognitive prompts to optimise task focus and affective cues to foster positive dynamics. In a controlled study (N = 48), we compared Baseline, Cognitive Feedback (CF), Affective Feedback (AF) and Combined Feedback (CAF) conditions. Results show CAF significantly improved task performance, reduced mental demand and enhanced social presence, outperforming all other conditions. Participants rated CAF as having the highest level of facilitator expertise and preference. These findings highlight the benefits of integrated AI-driven facilitation, offering design insights for human-centred, effective virtual collaboration tools that balance task efficiency with positive socio-emotional engagement.

Mixed Reality Agents (MiRAs) have been extensively studied to enhance virtual-physical interactions, using their ability to exist in both virtual and physical environments. However, little research has focused on enhancing perceived empathy in MiRAs, despite its potential for agent-assisted therapy, education, and training. To fill this gap, we investigate the impact of an Empathic Mixed Reality agent (EMiRA) that adapts to users' physiological states and physical events in a shooting game. We found that this adaptation enhanced users' social perceptions of the agent, including social presence, social connectedness, and perceived empathy. Physiological adaptation increased paternalism and reduced user dominance, while physical adaptation had no such effect. We discuss these findings and provide design implications for future EMiRAs.

Non-verbal cues like locomotion and posture influence users’ perceptions of Mixed Reality Agents (MiRAs). While Electroencephalography (EEG) captures cognitive responses, the influence of MiRAs’ locomotion and postures on brain activity remains underexplored. Additionally, few studies integrate subjective and behavioral measures with EEG to evaluate these cues’ impact on social perception. To address this, we conducted a within-subject study where participants played Gobang against three virtual agents in mixed reality: 1) a speech-only agent (S), 2) an embodied agent with speech and locomotion (S + L), and 3) an embodied agent with speech, locomotion, and posture (S + L + P). Results showed the S + L + P agent had higher engagement measured by the questionnaire but a lower EEG-based engagement index at AF3 than the S + L agent. Besides, the S + L + P was also rated higher in social presence, engagement, and emotional arousal than the S condition; No behavioral differences were observed. We discuss how MiRAs’ locomotion and posture affect users’ social perception and provide design implications for future human-agent interactions.

Understanding the relationship between cognitive load and affective state in remote work is vital for designing intuitive collaboration. We present CoAffinity, a multimodal dataset encompassing eight structured remote-work tasks, during which 39 participants provided self-reported measures (arousal, valence, positive/negative affect, and cognitive-load) while being recorded via audio, video, and physiological signals (PPG and GSR). Spanning over 38 hours of annotated data, our approach involved precise timestamp alignment, short and long-session labelling, and subsequent machine-learning and deep-learning benchmarks. Key findings show that integrating multiple modalities, especially physiological data, significantly improves the detection of cognitive load and emotion, while group synchrony metrics highlight how physiological coherence shifts under varied task demands. By capturing complex cognitive-emotional dynamics in realistic remote settings, CoAffinity aims to advance affective computing, inform human-computer interaction research, and foster more empathetic remote collaboration tools

In the past decade, there has been significant advancement in designing wearable neural interfaces for controlling neurorobotic systems, particularly bionic limbs. These interfaces function by decoding signals captured noninvasively from the skin’s surface. Portable high-density surface electromyography (HD-sEMG) modules combined with deep learning decoding have attracted interest by achieving excellent gesture prediction and myoelectric control of prosthetic systems and neurorobots. However, factors like small electrode size and unstable electrode-skin contacts make HD-sEMG susceptible to pixel electrode drops. The sparse electrode-skin disconnections rooted in issues such as low adhesion, sweating, hair blockage, and skin stretch challenge the reliability and scalability of these modules as the perception unit for neurorobotic systems. This paper proposes a novel deep-learning model providing resiliency for HD-sEMG modules, which can be used in the wearable interfaces of neurorobots. The proposed 3D Dilated Efficient CapsNet model trains on an augmented input space to computationally ‘force’ the network to learn channel dropout variations and thus learn robustness to channel dropout. The proposed framework maintained high performance under a sensor dropout reliability study conducted. Results show conventional models’ performance significantly degrades with dropout and is recovered using the proposed architecture and the training paradigm.

In human-agent interaction, establishing trust and a social bond with the agent is crucial to improving communication quality and performance in collaborative tasks. This paper investigates how a Mixed Reality Agent’s (MiRA) ability to acknowledge a user’s physiological state affects perceptions such as empathy, social connectedness, presence, and trust. In a within-subject study with 24 subjects, we varied the companion agent’s awareness during a mixed-reality first-person shooting game. Three agents provided feedback based on the users’ physiological states: (1) No Awareness Agent (NAA), which did not acknowledge the user’s physiological state; (2) Random Awareness Agent (RAA), offering feedback with varying accuracy; and (3) Accurate Awareness Agent (AAA), which provided consistently accurate feedback. Subjects reported higher scores on perceived empathy, social connectedness, presence, and trust with AAA compared to RAA and NAA. Interestingly, despite exceeding NAA in perception scores, RAA was the least favored as a companion. The findings and implications for the design of MiRA interfaces are discussed, along with the limitations of the study and directions for future work.

In collaborative settings where multiple individuals are tasked with completing a shared goal, understanding one’s partner’s emotional state could be crucial for achieving a successful outcome. This is particularly relevant in remote collaboration contexts, where physical distance can impede understanding, empathy, and mutual comprehension between partners. In this paper, we demonstrate representing emotional patterns from physiological data in a shared Virtual Reality (VR) environment, and explore how it impacted communication styles. A user study investigated the potential effects of this emotional representation in fostering empathetic communication during remote collaboration. The study’s findings revealed that although there was minimal variance in the workload associated with observing physiological cues, participants generally preferred monitoring their partner’s attentional state. However, with the assembly task chosen, most participants only directed a minimal proportion of their attention toward the physiological cues displayed by their partner, and were frequently uncertain of how to interpret and use the information obtained. We also discuss limitations of the research and opportunities for future work.

There is little research on how Virtual Reality (VR) applications can identify and respond meaningfully to users' emotional changes. In this paper, we investigate the impact of Context-Aware Empathic VR (CAEVR) on the emotional and cognitive aspects of user experience in VR. We developed a real-time emotion prediction model using electroencephalography (EEG), electrodermal activity (EDA), and heart rate variability (HRV) and used this in personalized and generalized models for emotion recognition. We then explored the application of this model in a context-aware empathic (CAE) virtual agent and an emotion-adaptive (EA) VR environment. We found a significant increase in positive emotions, cognitive load, and empathy toward the CAE agent, suggesting the potential of CAEVR environments to refine user-agent interactions. We identify lessons learned from this study and directions for future work.

With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner’s mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and co-presence to an audio-only baseline. In addition, we tracked the musicians’ physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.

The volume rendering step used in Neural Radiance Fields (NeRFs) produces highly photorealistic results, but is inherently slow because it evaluates an MLP at a large number of sample points per ray. Previous work has addressed this by either proposing neural scene representations that are faster to evaluate or by pre-computing (and approximating) scene properties to reduce render times. In this work, we propose MCNeRF, a general Monte Carlo-based rendering algorithm that can speed up any NeRF representation. We show that the NeRF volume rendering integral can be efficiently computed via Monte Carlo integration using an importance sampling scheme based on ray density distributions. This allows us to use a small number of MLP evaluations to estimate pixel radiance. These noisy Monte Carlo estimates can be further denoised using an inexpensive image-space denoiser trained per-scene. We demonstrate that MCNeRF can be used to speed up NeRF representations like TensoRF by 7 × while closely matching their visual quality and without making the scene approximations that real-time NeRF rendering methods usually make.

In this paper, we share real-time collaborative gaze behaviours, hand pointing, gesturing, and heart rate visualisations between remote collaborators using a live 360 ° panoramic-video based Mixed Reality (MR) system. We first ran a pilot study to explore visual designs to combine communication cues with biofeedback (heart rate), aiming to understand user perceptions of empathic collaboration. We then conducted a formal study to investigate the effect of modality (Gaze+Hand, Hand-only) and interface (Near-Gaze, Embodied). The results show that the Gaze+Hand modality in a Near-Gaze interface is significantly better at reducing task load, improving co-presence, enhancing understanding and tightening collaborative behaviours compared to the conventional Embodied hand-only experience. Ranked as the most preferred condition, the Gaze+Hand in Near-Gaze condition is perceived to reduce the need for dividing attention to the collaborator’s physical location, although it feels slightly less natural compared to the embodied visualisations. In addition, the Gaze+Hand conditions also led to more joint attention and less hand pointing to align mutual understanding. Lastly, we provide a design guideline to summarize what we have learned from the studies on the representation between modality, interface, and biofeedback.

This paper introduces a framework designed to accurately predict piecewise linear mappings of arbitrary meshes via a neural network, enabling training and evaluating over heterogeneous collections of meshes that do not share a triangulation, as well as producing highly detail-preserving maps whose accuracy exceeds current state of the art. The framework is based on reducing the neural aspect to a prediction of a matrix for a single given point, conditioned on a global shape descriptor. The field of matrices is then projected onto the tangent bundle of the given mesh, and used as candidate jacobians for the predicted map. The map is computed by a standard Poisson solve, implemented as a differentiable layer with cached pre-factorization for efficient training. This construction is agnostic to the triangulation of the input, thereby enabling applications on datasets with varying triangulations. At the same time, by operating in the intrinsic gradient domain of each individual mesh, it allows the framework to predict highly-accurate mappings. We validate these properties by conducting experiments over a broad range of scenarios, from semantic ones such as morphing, registration, and deformation transfer, to optimization-based ones, such as emulating elastic deformations and contact correction, as well as being the first work, to our knowledge, to tackle the task of learning to compute UV parameterizations of arbitrary meshes. The results exhibit the high accuracy of the method as well as its versatility, as it is readily applied to the above scenarios without any changes to the framework.

Meshes are important representations of physical 3D entities in the virtual world. Applications like rendering, simulations and 3D printing require meshes to be manifold so that they can interact with the world like the real objects they represent. Prior methods generate meshes with great geometric accuracy but poor manifoldness. In this work, we propose NeuralMeshFlow (NMF) to generate two-manifold meshes for genus-0 shapes. Specifically, NMF is a shape auto-encoder consisting of several Neural Ordinary Differential Equation (NODE)(1) blocks that learn accurate mesh geometry by progressively deforming a spherical mesh. Training NMF is simpler compared to state-of-the-art methods since it does not require any explicit mesh-based regularization. Our experiments demonstrate that NMF facilitates several applications such as single-view mesh reconstruction, global shape parameterization, texture mapping, shape deformation and correspondence. Importantly, we demonstrate that manifold meshes generated using NMF are better-suited for physically-based rendering and simulation compared to prior works.

With the advancement of Artificial Intelligence technology to make smart devices, understanding how humans develop trust in virtual agents is emerging as a critical research field. Through our research, we report on a novel methodology to investigate user’s trust in auditory assistance in a Virtual Reality (VR) based search task, under both high and low cognitive load and under varying levels of agent accuracy. We collected physiological sensor data such as electroencephalography (EEG), galvanic skin response (GSR), and heart-rate variability (HRV), subjective data through questionnaire such as System Trust Scale (STS), Subjective Mental Effort Questionnaire (SMEQ) and NASA-TLX. We also collected a behavioral measure of trust (congruency of users’ head motion in response to valid/ invalid verbal advice from the agent). Our results indicate that our custom VR environment enables researchers to measure and understand human trust in virtual agents using the matrices, and both cognitive load and agent accuracy play an important role in trust formation. We discuss the implications of the research and directions for future work.

We present AffectivelyVR, a personalized real-time emotion recognition system in Virtual Reality (VR) that enables an emotion-adaptive virtual environment. We used off-the-shelf Electroencephalogram (EEG) and Galvanic Skin Response (GSR) physiological sensors to train user-specific machine learning models while exposing users to affective 360° VR videos. Since emotions are largely dependent on interpersonal experiences and expressed in different ways for different people, we personalize the model instead of generalizing it. By doing this, we achieved an emotion recognition rate of 96.5% using the personalized KNN algorithm, and 83.7% using the generalized SVM algorithm.

Human trust is a psycho-physiological state that is difficult to measure, yet is becoming increasingly important for the design of human-computer interactions. This paper explores if human trust can be measured using physiological measures when interacting with a computer interface, and how it correlates with cognitive load. In this work, we present a pilot study in Virtual Reality (VR) that uses a multi-sensory approach of Electroencephalography (EEG), galvanic skin response (GSR), and Heart Rate Variability (HRV) to measure trust with a virtual agent and explore the correlation between trust and cognitive load. The goal of this study is twofold; 1) to determine the relationship between biosignals, or physiological signals with trust and cognitive load, and 2) to introduce a pilot study in VR based on cognitive load level to evaluate trust. Even though we could not report any significant main effect or interaction of cognitive load and trust from the physiological signal, we found that in low cognitive load tasks, EEG alpha band power reflects trustworthiness on the agent. Moreover, cognitive load of the user decreases when the agent is accurate regardless of task’s cognitive load. This could be possible because of small sample size, tasks not stressful enough to induce high cognitive load due to lab study and comfortable environment or timestamp synchronisation error due to fusing data from various physiological sensors with different sample rate.

We present results from research exploring the effect of sharing virtual gaze and pointing cues in a wearable interface for remote collaboration. A local worker wears a Head-mounted Camera, Eye-tracking camera and a Head-Mounted Display and shares video and virtual gaze information with a remote helper. The remote helper can provide feedback using a virtual pointer on the live video view. The prototype system was evaluated with a formal user study. Comparing four conditions, (1) NONE (no cue), (2) POINTER, (3) EYE-TRACKER and (4) BOTH (both pointer and eye-tracker cues), we observed that the task completion performance was best in the BOTH condition with a significant difference of POINTER and EYETRACKER individually. The use of eye-tracking and a pointer also significantly improved the co-presence felt between the users. We discuss the implications of this research and the limitations of the developed system that could be improved in further work.

Performance engineering is a badly needed skill for implementing and running IT systems, but performance engineers are hard to find in the market. This paper presents our experiences in rolling out training and certification in a first level course on performance engineering across a large enterprise. We present data and lessons learned on the nominations for the rollout, the design and analysis of theory and practical exams, and the methods used to ensure fairness in a rollout spanning hundreds of nominations. We present results that show trainees to perform exceeding well much against conventional wisdom, and we also show how the success of the rollout has led to a number of beneficial initiatives for the company.