Cross-functional teams struggle when static collaboration tools fail to keep pace with dynamic conversations. Through a formative study with seven professionals, we identified a critical gap: designers and developers speak different vocabularies, causing semantic misalignments. We present Cognitive Bridge, an AI system that monitors multimodal cues (facial expressions, speech, workspace activity) to detect emerging misunderstandings, then generates adaptive boundary objects, visual diagrams, wireframes, and flowcharts that translate between professional perspectives in real-time. Our controlled study with 16 designer-developer dyads found that Cognitive Bridge reduced communication conflicts by 47% and increased implementable solutions by 34% compared to baseline tools. However, analysis revealed a solution-exploration tradeoff: while AI accelerated alignment, it risked premature convergence that constrained creative exploration. We contribute: (1) a novel system for AI-generated boundary objects, and (2) design implications for balancing cognitive scaffolding with creative agency preservation.

Digital health has strong potential for promoting physical activity (PA), yet interventions often fail to sustain engagement among culturally and linguistically diverse (CALD) women. Prior reviews focus on short-term efficacy or surface-level localisation, while a design-oriented synthesis of deep cultural adaptation and long-term strategies remain limited. This scoping review systematically screened 1968 records, analysed 18 studies and identified a critical design paradox: techno-solutionist systems overlook social and cultural barriers, while social-support features often fail in low-activity social networks. To address this gap, we propose the Culturally Embedded Interaction Framework, integrating five dimensions: culturally-grounded measurement, multi-modal interaction, contextual and temporal adaptability, embedded social weaving, and theory-guided cultural adaptation. The framework advances beyond accessibility-focused approaches by mapping behavioural theory to design mechanisms that support sustained and culturally plural participation. We provide actionable design principles to help HCI researchers and practitioners move from one-size-fits-all models toward adaptive, theory-informed, and culturally sustaining design.

Video conferencing is essential for remote collaboration, but it often leads to fatigue, reduced social presence and ineffective communication. Traditional human facilitators can address these challenges but cannot scale to meet the demands of countless daily virtual meetings across organisations. To address these challenges, we introduce Cognitive Load and Affect Aware Agent (CLARA), an AI-mediated facilitator that enhances group decision-making by dynamically managing cognitive load and affective engagement. CLARA employs real-time multimodal assessment of group states, providing adaptive cognitive prompts to optimise task focus and affective cues to foster positive dynamics. In a controlled study (N = 48), we compared Baseline, Cognitive Feedback (CF), Affective Feedback (AF) and Combined Feedback (CAF) conditions. Results show CAF significantly improved task performance, reduced mental demand and enhanced social presence, outperforming all other conditions. Participants rated CAF as having the highest level of facilitator expertise and preference. These findings highlight the benefits of integrated AI-driven facilitation, offering design insights for human-centred, effective virtual collaboration tools that balance task efficiency with positive socio-emotional engagement.

To enhance focused eating and dining socialization, previous Human-Food Interaction research has indicated that external devices can support these dining objectives and immersion. However, methods that focus on the food itself and the diners themselves have remained underdeveloped. In this study, we integrated biofeedback with food, utilizing diners' heart rates as a source of the food's appearance to promote focused eating and dining socialization. By employing LED lights, we dynamically displayed diners' real-time physiological signals through the transparency of the food. Results revealed significant effects on various aspects of dining immersion, such as awareness perceptions, attractiveness, attentiveness to each bite, and emotional bonds with the food. Furthermore, to promote dining socialization, we established a "Sharing Bio-Sync Food" dining system to strengthen emotional connections between diners. Based on these findings, we developed tableware that integrates biofeedback into the culinary experience.

Nowadays, touch remains essential for emotional conveyance and interpersonal communication as more interactions are mediated remotely. While many studies have discussed the effectiveness of using haptics to communicate emotions, incorporating affect into haptic design still faces challenges due to individual user tactile acuity and preferences. We assessed the conveying of emotions using a two-channel haptic display, emphasizing individual differences. First, 24 participants generated 187 haptic messages reflecting their immediate sentiments after watching 8 emotionally charged film clips. Afterwards, 19 participants were asked to identify emotions from haptic messages designed by themselves and others, yielding 593 samples. Our findings suggest potential links between haptic message decoding ability and emotional traits, particularly Emotional Competence (EC) and Affect Intensity Measure (AIM). Additionally, qualitative analysis revealed three strategies participants used to create touch messages: perceptive, empathetic, and metaphorical expression.

Social virtual reality (VR) is the experience of a shared virtual space populated by virtual representations of each individual, allowing them to communicate, collaborate and interact with each other not unlike the real world. Conventional avatars generally mirror factors like an individual’s appearance, speech, movement, and so on, yet a VR can offer many more possibilities to represent a person beyond what reality can offer. One such representation is that of emotions and empathy. To that regard, we propose Bioresponsive Avatars, an avatar system that predicts user emotional states and represents them visually via their avatar appearance. To achieve this, we first conducted an avatar design workshop to understand how user’s imagine emotional states to appear on an avatar. Then, we performed an in-the-wild demonstration of a social VR prototype where dyadic users are presented with affective topics to communicate while their avatars adapt based on their predicted emotions. • Developed bioresponsive avatars to map emotions into Valence–Arousal model. • Implemented a real-time avatar using physiological sensing and ML in social VR. • Conducted an in-the-wild study with 55 participants, noting patterns of empathy in VR.

Autistic children may often experience challenges in mastering daily living skills crucial for their independence and well-being. This study introduces “Tidd,” an augmented tabletop interactive system designed to assist autistic children in practising daily living skills in an engaging and physically interactive environment. We conducted a user study in a medical rehabilitation centre and an integrated kindergarten. Seventeen autistic children aged three to five years used Tidd in training sessions covering two vital skills: bed-making and dressing. Progress was evaluated through task completion and progress tracking, observational data for children, and therapist qualitative feedback. Therapists reported that Tidd was beneficial in maintaining the children’s attention and enhancing their motivation. Observational data further suggested increased engagement and decreased frustration during tasks. This study with Tidd highlights the potential of tabletop interaction to support therapists in training autistic children to learn daily living skills. • Tidd: AR tabletop device aiding autistic children in bed-making and dressing. • Improved task accuracy and engagement in study with 17 autistic children. • Therapists validated Tidd’s practicality and appeal in autism interventions.

Introspectus AI is a generative AI-based system designed to enhance self-reflection and support positive behavior change. By leveraging multimodal information from users' daily life recordings, it provides personalized and detailed feedback, aiming to deepen self-awareness and facilitate positive behavioral adjustments. This study explores the short-term and long-term impacts of interacting with Introspectus AI, focusing on its potential to enhance reflective practices and improve the acceptance of generative AI tools. Following the user experience was defined through an initial round of workshops with four experts. The resulting system was evaluated through a long-term study involving 64 participants. The results demonstrate that AI-supported interventions significantly improved engagement in self-reflection, the need for reflection, and insight, while also increasing user acceptance of generative AI over time. These findings underscore the potential of generative AI as a practical tool for self-improvement, offering insights into its broader applicability in promoting well-being and personal growth.

Understanding the relationship between cognitive load and affective state in remote work is vital for designing intuitive collaboration. We present CoAffinity, a multimodal dataset encompassing eight structured remote-work tasks, during which 39 participants provided self-reported measures (arousal, valence, positive/negative affect, and cognitive-load) while being recorded via audio, video, and physiological signals (PPG and GSR). Spanning over 38 hours of annotated data, our approach involved precise timestamp alignment, short and long-session labelling, and subsequent machine-learning and deep-learning benchmarks. Key findings show that integrating multiple modalities, especially physiological data, significantly improves the detection of cognitive load and emotion, while group synchrony metrics highlight how physiological coherence shifts under varied task demands. By capturing complex cognitive-emotional dynamics in realistic remote settings, CoAffinity aims to advance affective computing, inform human-computer interaction research, and foster more empathetic remote collaboration tools

Neurodivergent individuals, particularly those with Autism and Attention Deficit Hyperactivity Disorder (ADHD), frequently experience anxiety, panic attacks, meltdowns, and emotional dysregulation due to societal pressures and inadequate accommodations. These challenges are especially pronounced for neurodivergent women and non-binary individuals navigating intersecting barriers of neurological differences and gender expectations. This research investigates virtual reality (VR) as a portable safe space for emotional regulation, addressing challenges of sensory overload and motion sickness while enhancing relaxation capabilities. Our mixed-methods approach included an online survey (N=223) and an ideation workshop (N=32), which provided key design elements for creating effective calming VR environments. Based on these findings, we developed and iteratively tested VR prototypes with neurodivergent women and non-binary participants (N=12), leading to a final version offering enhanced adaptability to individual sensory needs. This final prototype underwent a comprehensive evaluation with 25 neurodivergent participants to assess its effectiveness as a regulatory tool. This research contributes to the development of inclusive, adaptive VR environments that function as personalized "portable silent rooms" offering neurodivergent individuals on-demand access to sensory regulation regardless of physical location.

To enhance the dining experience, prior studies in Human-Computer Interaction (HCI) and gastrophysics have demonstrated that modifying the static shape of solid foods can amplify taste perception. However, the exploration of dynamic shape-changing mechanisms in liquid foods remains largely untapped. In the present study, we employ cymatics, a scientific discipline focused on utilizing sound frequencies to generate patterns in liquids and particles—to augment the drinking experience. Utilizing speakers, we dynamically reshaped liquids exhibiting five distinct taste profiles and evaluated resultant changes in taste perception and drinking experience. Our research objectives extend beyond merely augmenting taste from visual to tactile sensations; we also prioritize the experiential aspects of drinking. Through a series of experiments and workshops, we revealed a significant impact on taste perception and overall drinking experience when mediated by cymatics effects. Building upon these findings, we designed and developed tableware to integrate cymatics principles into gastronomic experiences.

In collaborative settings where multiple individuals are tasked with completing a shared goal, understanding one’s partner’s emotional state could be crucial for achieving a successful outcome. This is particularly relevant in remote collaboration contexts, where physical distance can impede understanding, empathy, and mutual comprehension between partners. In this paper, we demonstrate representing emotional patterns from physiological data in a shared Virtual Reality (VR) environment, and explore how it impacted communication styles. A user study investigated the potential effects of this emotional representation in fostering empathetic communication during remote collaboration. The study’s findings revealed that although there was minimal variance in the workload associated with observing physiological cues, participants generally preferred monitoring their partner’s attentional state. However, with the assembly task chosen, most participants only directed a minimal proportion of their attention toward the physiological cues displayed by their partner, and were frequently uncertain of how to interpret and use the information obtained. We also discuss limitations of the research and opportunities for future work.

The limited nonverbal cues and spatially distributed nature of remote communication make it challenging for unacquainted members to be expressive during social interactions over video conferencing. Though it enables seeing others' facial expressions, the visual feedback can instead lead to unexpected self-focus, resulting in users missing cues for others to engage in the conversation equally. To support expressive communication and equal participation among unacquainted counterparts, we propose SealMates, a behavior-driven avatar in which the avatar infers the engagement level of the group based on collective gaze and speech patterns and then moves across interlocutors' windows in the video conferencing. By conducting a controlled experiment with 15 groups of triads, we found the avatar's movement encouraged people to experience more self-disclosure and made them perceive everyone was equally engaged in the conversation than when there was no behavior-driven avatar. We discuss how a behavior-driven avatar influences distributed members' perceptions and the implications of avatar-mediated communication for future platforms.

We introduce RadarHand, a wrist-worn wearable with millimetre wave radar that detects on-skin touch-based proprioceptive hand gestures. Radars are robust, private, small, penetrate materials, and require low computation costs. We first evaluated the proprioceptive and tactile perception nature of the back of the hand and found that tapping on the thumb is the least proprioceptive error of all the finger joints, followed by the index finger, middle finger, ring finger, and pinky finger in the eyes-free and high cognitive load situation. Next, we trained deep-learning models for gesture classification. We introduce two types of gestures based on the locations of the back of the hand: generic gestures and discrete gestures. Discrete gestures are gestures that start at specific locations and end at specific locations at the back of the hand, in contrast to generic gestures, which can start anywhere and end anywhere on the back of the hand. Out of 27 gesture group possibilities, we achieved 92% accuracy for a set of seven gestures and 93% accuracy for the set of eight discrete gestures. Finally, we evaluated RadarHand’s performance in real-time under two interaction modes: Active interaction and Reactive interaction. Active interaction is where the user initiates input to achieve the desired output, and reactive interaction is where the device initiates interaction and requires the user to react. We obtained an accuracy of 87% and 74% for active generic and discrete gestures, respectively, as well as 91% and 81.7% for reactive generic and discrete gestures, respectively. We discuss the implications of RadarHand for gesture recognition and directions for future works.

There is little research on how Virtual Reality (VR) applications can identify and respond meaningfully to users' emotional changes. In this paper, we investigate the impact of Context-Aware Empathic VR (CAEVR) on the emotional and cognitive aspects of user experience in VR. We developed a real-time emotion prediction model using electroencephalography (EEG), electrodermal activity (EDA), and heart rate variability (HRV) and used this in personalized and generalized models for emotion recognition. We then explored the application of this model in a context-aware empathic (CAE) virtual agent and an emotion-adaptive (EA) VR environment. We found a significant increase in positive emotions, cognitive load, and empathy toward the CAE agent, suggesting the potential of CAEVR environments to refine user-agent interactions. We identify lessons learned from this study and directions for future work.

Dementia describes a syndrome of cognitive degeneration, and Behavioural and Psychological Symptoms of Dementia (BPSD) is the non-cognitive symptom. BPSD can be improved by care services. To aid better care service, we explore the potential of using Augmented Reality (AR) to support dementia education for medical workers in three steps: (1) We explore medical workers’ perspective on dementia care lived experience and XR, (2) we co-design an educational experience containing an AR-based application and a 5-min activity with medical workers, (3) we evaluate the effectiveness of the system through a mixed method study. Our result shows that the AR experience successfully touches participants, and motivates them to reflect on the provision of care service. On this basis, we discuss the elements and challenges of designing XR-enabled dementia education for users unfamiliar with novel technology, and the potential of using XR in clinical education.

As the population ages, many will acquire visual impairments. To improve design for these users, it is essential to build awareness of their perspective during everyday routines, especially for design students.

With voice user interfaces (VUIs) becoming ubiquitous and speech synthesis technology maturing, it is possible to synthesise voices to resemble our friends and relatives (which we will collectively call 'kin') and use them on VUIs. However, designing such interfaces and investigating how the familiarity of kin voices affect user perceptions remain under-explored. Our surveys and interviews with 25 users revealed that VUIs using kin voices were perceived as more engaging, persuasive and safer yet eerier than VUIs using common virtual assistant voices. We then developed a technology probe, KinVoice, an Alexa-based VUI that was deployed in three households over two weeks. Users set reminders using KinVoice, which in turn, gave the reminders in synthesised kin voices. This was to explore users' needs, uncover challenges involved and inspire new applications. We discuss design guidelines for integrating familiar kin voices into VUIs, applications that benefit from its usage, and implications for balancing voice realism and usability with security and diversification.

The tactile and kinesthetic sensation of pushing a button is usually lost when interacting with modern devices like touchscreens and/or virtual reality platforms. We present FingerFlex, a standalone glove wearable actuating the metacarpophalangeal joint (MCP) of each finger via shape memory alloy (SMA). SMA actuation is subtle, silent, and light, making it ideal for actuation of the fingers which we use to simulate the sensation of pressing a button.

With the advancement of Artificial Intelligence technology to make smart devices, understanding how humans develop trust in virtual agents is emerging as a critical research field. Through our research, we report on a novel methodology to investigate user’s trust in auditory assistance in a Virtual Reality (VR) based search task, under both high and low cognitive load and under varying levels of agent accuracy. We collected physiological sensor data such as electroencephalography (EEG), galvanic skin response (GSR), and heart-rate variability (HRV), subjective data through questionnaire such as System Trust Scale (STS), Subjective Mental Effort Questionnaire (SMEQ) and NASA-TLX. We also collected a behavioral measure of trust (congruency of users’ head motion in response to valid/ invalid verbal advice from the agent). Our results indicate that our custom VR environment enables researchers to measure and understand human trust in virtual agents using the matrices, and both cognitive load and agent accuracy play an important role in trust formation. We discuss the implications of the research and directions for future work.

Research in sociology shows that effective conversation relates to people’s spatial and orientational relationship, namely the proxemics (distance, eye contact, synchrony) and the F-formation (orientation and arrangement). In this work, we introduce novel conversational paradigms that effects conventional F-formation by introducing the concept of multi-directional conversation. Multiplex Vision is a head-mounted device capable of providing a 360° field-of-view (FOV) and facilitating multi-user interaction multi-directionally, thereby providing novel methods on how people can interact with each other. We propose 3 possible new forms of interactions from our prototype: one-to-one, one-to-many, and many-to-many. To facilitate them, we manipulate 2 key variables, which are the viewing parameter and the display parameter. To gather feedback for our system, we conducted a study to understand information transfer between various modes, as well as a user study on how different proposed paradigms effect conversation. Finally, we discuss present and future use cases that can benefit from our system.

We present AffectivelyVR, a personalized real-time emotion recognition system in Virtual Reality (VR) that enables an emotion-adaptive virtual environment. We used off-the-shelf Electroencephalogram (EEG) and Galvanic Skin Response (GSR) physiological sensors to train user-specific machine learning models while exposing users to affective 360° VR videos. Since emotions are largely dependent on interpersonal experiences and expressed in different ways for different people, we personalize the model instead of generalizing it. By doing this, we achieved an emotion recognition rate of 96.5% using the personalized KNN algorithm, and 83.7% using the generalized SVM algorithm.

Reading in public spaces can often be tricky if we wish to keep the contents away from the prying eye. We propose Private Reader, an eye-tracking approach towards maintaining privacy while reading by rendering only the portion of text that is gazed by the reader. We conducted a user study by evaluating for both the reader and observer in terms of privacy, reading comfort, and reading speed for three reading modes; normal, underscored, and scrambled text. "Scrambled" performs best in terms of perceived effort and frustration for the shoulder surfer. Our contribution is threefold; we developed a system to preserve privacy by rendering only the text at gaze-point of the reader, we conducted a user study to evaluate user preferences and subjective task load, and we suggested several scenarios where Private Reader is useful in public spaces.

Human trust is a psycho-physiological state that is difficult to measure, yet is becoming increasingly important for the design of human-computer interactions. This paper explores if human trust can be measured using physiological measures when interacting with a computer interface, and how it correlates with cognitive load. In this work, we present a pilot study in Virtual Reality (VR) that uses a multi-sensory approach of Electroencephalography (EEG), galvanic skin response (GSR), and Heart Rate Variability (HRV) to measure trust with a virtual agent and explore the correlation between trust and cognitive load. The goal of this study is twofold; 1) to determine the relationship between biosignals, or physiological signals with trust and cognitive load, and 2) to introduce a pilot study in VR based on cognitive load level to evaluate trust. Even though we could not report any significant main effect or interaction of cognitive load and trust from the physiological signal, we found that in low cognitive load tasks, EEG alpha band power reflects trustworthiness on the agent. Moreover, cognitive load of the user decreases when the agent is accurate regardless of task’s cognitive load. This could be possible because of small sample size, tasks not stressful enough to induce high cognitive load due to lab study and comfortable environment or timestamp synchronisation error due to fusing data from various physiological sensors with different sample rate.

We propose PanoFlex, an adaptive method for projecting panoramic vision using a dynamic distortion method based on eye gaze. We stream real-time video from a 360° camera and project the view on a plane to the user. The user controls the distortion of this equirectangular projection using eye gaze. For our first user study, we compare our method with conventional equirectangular projection considering the impact on spatial perception. For our second study, we perform a simulator sickness evaluation when the user performs regular daily activities. We found that PanoFlex did not carry any significant negative impact towards the user’s spatial perception, perceived task load, and simulator sickness compared to the more conventional equirectangular view.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

Gaze-based interactions promise to be fast, intuitive and effective in controlling virtual and augmented environments. Yet, there is still a lack of usable 3D navigation and observation techniques. In this work: 1) We introduce a highly advantageous orbital navigation technique, AnyOrbit, providing an intuitive and hands-free method of observation in virtual environments that uses eye-tracking to control the orbital center of movement; 2) The versatility of the technique is demonstrated with several control schemes and use-cases in virtual/augmented reality head-mounted-display and desktop setups, including observation of 3D astronomical data and spectator sports.

Navigation and mobility mechanics for virtual environments aim to be realistic or fun, but rarely prioritize the accuracy of movement. We propose PinchMove, a highly accurate navigation mechanic utilizing pinch gestures and manipulation of the viewport for confined environments that prefers accurate movement. We ran a pilot study to first determine the degree of simulator sickness caused by this mechanic, and a comprehensive user study to evaluate its accuracy in a virtual environment. We found that utilizing an 80° tunneling effect at a maximum speed of 15.18° per second was deemed suitable for PinchMove in reducing motion sickness. We also found our system to be at average, more accurate in enclosed virtual environments when compared to conventional methods. This paper makes the following three contributions: 1) We propose a navigation solution in near-field virtual environments for accurate movement, 2) we determined the appropriate tunneling effect for our method to minimize motion sickness, and 3) We validated our proposed solution by comparing it with conventional navigation solutions in terms of accuracy of movement. We also propose several use- case scenarios where accuracy in movement is desirable and further discuss the effectiveness of PinchMove.

Nonverbal cues, especially eye gaze, plays an important role in our daily communication, not just as an indicator of interest, but also as a method to convey information to another party. In this work, we propose a simulation of human eye gaze in Virtual Reality content to improve immersion of interaction between user and virtual agent. We developed an eye-tracking integrated interactive narrative content with a focus on player's interaction with gaze aware virtual agent, which is capable of reacting towards the player's gaze to simulate real human-to-human communication in VR environment and conducted an initial study to measure user's reaction.

Navigating in a natural way in augmented reality (AR) and virtual reality (VR) spaces is a large challenge. To this end, we present ArmSwingVR, a locomotion solution for AR/VR spaces that preserves immersion, while being low profile compared to current solutions, particularly walking-in-place (WIP) methods. The user simply needs to swing their arms naturally to navigate in the direction where the arms are swung, without any feet or head movement. The benefits of ArmSwingVR are that arm swinging feels natural for bipedal organisms second only to leg movement, no additional peripherals or sensors are required, it is less obtrusive to swing our arms as opposed to WIP methods, and requires less energy allowing prolong uses for AR/VR. A conducted user study found that our method does not sacrifice immersion while also being more low profile and less energy consumption compared to WIP.