Extended reality (XR) technologies offer significant potential to create immersive virtual companionship experiences that support social connection, emotional engagement, and everyday practical needs. However, little is known about how people envision day-to-day interactions with virtual companions in XR environments, raising questions about how they should be designed. To address this gap, we conducted speculative design workshops with 16 participants experienced in AI and XR, generating 16 diverse design concepts spanning varied contexts, use cases and XR-specific affordances. Our analysis reveals key opportunities and emerging challenges in designing virtual companions for immersive environments, demonstrating how they foster meaningful companionship while supporting everyday activities. We further uncover a social tension around companions’ embodied presence and visibility in shared spaces: users often prefer private, controllable interactions (engaging with a companion only they can see), which can conflict with social norms, create ambiguity for bystanders, or result in missing context during co-present interactions.

Assessing fairness in artificial intelligence (AI) typically involves AI experts who select protected features, fairness metrics, and set fairness thresholds to assess outcome fairness. However, little is known about how stakeholders, particularly those affected by AI outcomes but lacking AI expertise, assess fairness. To address this gap, we conducted a qualitative study with 26 stakeholders without AI expertise, representing potential decision subjects in a credit rating scenario, to examine how they assess fairness when placed in the role of deciding on features with priority, metrics, and thresholds. We reveal that stakeholders' fairness decisions are more complex than typical AI expert practices: they considered features far beyond legally protected features, tailored metrics for specific contexts, set diverse yet stricter fairness thresholds, and even preferred designing customized fairness. Our results extend the understanding of how stakeholders can meaningfully contribute to AI fairness governance and mitigation, underscoring the importance of incorporating stakeholders' nuanced fairness judgments.

This paper proposes a novel approach for the dynamic generation of training scenarios in Virtual Reality (VR), in the context of first-responders training. Our proposed model leverages an ontology, Large Language Models (LLMs) and Fuzzy Cognitive Maps (FCM) in order to produce scenarios meant to train first responders emotion regulation skills to be better prepared for complex, high-risk situations. These scenarios take into account users’ personal profiles, containing notably individual emotional vulnerabilities, i.e. which situational elements are susceptible to stress them more than others, in order to produce personalized, effective training experiences. The Fuzzy Cognitive Maps allow the system to model a degree of uncertainty when choosing story events, with probabilities for choosing the next scenario event being influenced by several factors, mainly the user’s profile and the virtual world state. Our evaluation shows that for randomized initial situations and user profiles, our system generates varied story events, offering diversity in generated scenarios.

Immersive realities enable social interactions that are radically different from traditional communication technologies, but how we experience immersion together is not yet indistinguishable from face-to-face interactions. Some social signals are not stable across realities, may change in semantics, or are missing all together. Understanding how social signals impact behaviours and experiences of social connection in immersive environments is key to creating experiences that are meaningful, satisfying, and productive. We completed a lab study where 6 groups of 6 participants (N=36) completed co-located social tasks in an instrumented face-to-face environment and its digital twin, creating a rich open dataset of 1.8 million rows across 45 columns. Our quantitative results demonstrate the stability of position as a social signal, measure lower social synchronisation in XR compared to face-to-face, and propose a method for bench marking XR against face-to-face interactions. This enables direct quantitative comparisons between experiences of co-located physical and virtual interactions for the first time.

International audience

For efficient human-agent interaction, an agent should proactively recognize their target user and prepare for upcoming interactions. We formulate this challenging problem as a novel task of jointly forecasting a person’s intent to interact with the agent, their attitude towards the agent and the action they will perform, from the agent’s (egocentric) perspective. We propose SocialEgoNet - a graph-based spatiotemporal framework that exploits task dependencies through a hierarchical multitask learning approach. SocialEgoNet uses whole-body skeletons (keypoints from face, hands and body) extracted from only 1 second of video input for high inference speed. For evaluation, we augment an existing egocentric human-agent interaction dataset with new class labels and bounding box annotations. Extensive experiments on this augmented dataset, named JPL-Social, demonstrate real-time inference and superior performance (average accuracy across all tasks: 83.15%) of our model outperforming several competitive baselines. The additional annotations and code are available at github.com/biantongfei/SocialEgoNet.

Foundation Models trained to perform a certain task can be fine-tuned to other tasks with limited data and computational resources. The advantage of such practice is that it makes it possible to benefit, at least indirectly, from the large amounts of data and the major computational infrastructure necessary for training a Foundation Model. However, there is a limitation too, namely that the few organizations that have the major resources necessary to develop and train Foundation Models do it only for the modalities that are of interest to them. For this reason, this article proposes to fine-tune Foundation Models trained on speech and photoplethysmography signals to perform stress detection based on Electro-Dermal Activity, a modality for which no Foundation Model exists. To the best of our knowledge, this is one of the first works proposing experiments of this type and the results show state-of-the-art stress detection performances over a publicly available benchmark, even if speech and photoplethysmograhpy data differ significantly from Electro-Dermal Activity signals.

There is a growing need for social robots and intelligent agents that can effectively interact with and support users. For the interactions to be seamless, the agents need to analyse social scenes and behavioural cues from their (robot’s) perspective. Works that model human-agent interactions in social situations are few; and even those existing ones are computationally too intensive to be deployed in real time or perform poorly in real-world scenarios when only limited information is available. We propose a knowledge distillation framework that models social interactions through various multimodal cues, and yet is robust against incomplete and noisy information during inference. We train a teacher model with multimodal input (body, face and hand gestures, gaze, raw images) that transfers knowledge to a student model which relies solely on body pose. Extensive experiments on two publicly available human-robot interaction datasets demonstrate that our student modelachievesan average accuracy gain of 14.75% over competitive baselines on multiple downstream social understanding tasks, even with up to 51% of its input being corrupted. The student model is also highly efficient - less than 1% in size of the teacher model in terms of parameters and its latency is 11.9% of the teacher model.

In high-stakes collaborative situations, a decline in collaboration quality can lead to adverse events with significant consequences. Analyses performed by Human factor (HF) specialists, while effective in identifying and addressing collaboration issues, are case-specific and most of the time performed a posteriori. To address these limitations, our research focuses on a real-time assessment of collaboration processes using multimodal signals collected and analyzed during the activity. Existing collaboration profiles taxonomies face limitations such as a posteriori profiles detection and the absence of quantitative behavioral indicators that can be measured during the activity. Leveraging Virtual Reality (VR), we have developed a framework for evaluating collaboration in controlled setting, testing the effectiveness of a subset of multimodal signals to detect collaboration profiles. We test our approach in a study including 11 stereotyped collaborative scenarios applied to a VR puzzle-solving task. This study reveals the effectiveness of our approach in distinguishing between non-collaborative and highly collaborative profiles. However, challenges arise in discriminating between closely related collaborative profiles. This paper also proposes some guidelines on how to improve the collaboration profile detection framework and address other collaborative situations.

Numerous fairness metrics have been proposed and employed by artificial intelligence (AI) experts to quantitatively measure bias and define fairness in AI models. Recognizing the need to accommodate stakeholders' diverse fairness understandings, efforts are underway to solicit their input. However, conveying AI fairness metrics to stakeholders without AI expertise, capturing their personal preferences, and seeking a collective consensus remain challenging and underexplored. To bridge this gap, we propose a new framework, EARN ( Explain, Ask, Review, and Negotiate ) Fairness, which facilitates collective metric decisions among stakeholders without requiring AI expertise. The framework features an adaptable interactive system and a stakeholder-centered EARN Fairness process to Explain fairness metrics, Ask stakeholders' personal metric preferences, Review metrics collectively, and Negotiate a consensus on metric selection. To gather empirical results, we applied the framework to a credit rating scenario and conducted a user study involving 18 decision subjects without AI knowledge. We elicited their personal metric preferences and subsequently we studied how they reached metric consensus in team sessions. Our work shows that the EARN Fairness framework supports stakeholders to express and negotiate fairness preferences, and we provide practical guidance for implementing human-centered AI fairness in high-risk contexts. Through this approach, we aim to reach consensus of fairness perspectives, fostering more equitable and inclusive AI fairness.

Harassment in Social Virtual Reality (SVR) is a growing concern. The current SVR landscape features inconsistent access to non-standardised safety features, with minimal empirical evidence on their real-world effectiveness, usage and impact. We examine the use and effectiveness of safety tools across 12 popular SVR platforms by surveying 100 users about their experiences of different types of harassment and their use of features like muting, blocking, personal spaces and safety gestures. While harassment remained common-including hate speech, virtual stalking, and physical harassment-many find safety features insufficient or inconsistently applied. Reactive tools like muting and blocking are widely used, largely driven by users' familiarity from other platforms. Safety tools are also used to proactively curate individual virtual experiences, protecting users from harassment, but inadvertently leading to fragmented social spaces. We advocate for standardising proactive, rather than reactive, anti-harassment tools across platforms, and present insights into future safety feature development.

Our research investigates the impact of latency on presence and immersion in virtual reality (VR) environments, focusing on interactions with LLM-powered Embodied Conversational Agents (ECAs). We explore the effectiveness of multimodal feedback strategies—including filled pauses, nonverbal turn-taking behaviours, and visual feedback—in mitigating perceived latency. Eighteen participants were subjected to both a baseline condition, without feedback interventions, and a feedback-enhanced condition. Our findings indicate that the feedback condition significantly improved the sense of presence and immersion. We also found that perceived response time and users’ impressions of the agents improved, thereby increasing willingness for future interactions. Additionally, chatbot experience positively correlated with agent likeability, whereas VR experience showed no significant correlation. These results highlight the effectiveness of feedback modalities in enhancing spatial presence and overall immersion, despite latency issues in VR interactions with LLM-powered agents.

Public speaking is a near universally anxiety inducing social situation. Applications recreating social interactions with autonomous agents in virtual reality have been proposed as tools to alleviate public speaking anxiety, recreating exposure therapy methods with virtual audiences. To be efficient, such applications rely on the precise induction of controlled amounts of social stress. We reviewed the literature that studied the effect of virtual audience behaviors on participants’ stress levels, and found contradictory results that we attempt to explain in this article. We examine those studies and propose that social presence is one of the important factors mediating the effect of audience behavior on stress levels, and through social presence we can explain the different effects of audience behavior on stress observed in past studies. We conducted a study to test this theory, and expected that high social presence would lead to larger difference in stress induced by positive and negative audiences, as opposed to lower social presence which would attenuate the effect of audience behavior. We compared two display mediums, VR headsets to induce high social presence and wall projection to induce low social presence. We find an interaction effect of social presence and audience behavior on subjectively reported stress levels, but not on physiological measures.

This Workshop aims at stimulating multi-disciplinary discussions about the challenges related to corpus creation and annotation for social skills behavior analysis. Contributions from computational, psychological and psychometrics perspectives, as well as applications including platforms to share corpora and annotations, are welcomed. The main challenges related to corpus creation include the choice of the best setup and sensors, finding a trade-off between eliciting natural interactions, limiting invasiveness and collecting precise information. The second issue in this context regards the process of annotation. The choice of the type of annotators (experts vs. nonexperts), the type of annotations (automatic vs. manual, continue vs. discrete), the temporal segmentation (windowed vs. holistic) is crucial for a correct measure of the phenomenon of interest and getting significant results. The topics of CATS2021 will have a strong impact on researchers and stakeholders across different disciplines, such as Computer Science, Social Signal Processing, Psychology, Statistics. Leveraging the opportunities offered by such a multidisciplinary environment, the participants could enrich their perspective, strengthen their practices and methodologies and draw together a research roadmap tackling the discussed challenges, which might be taken up in future collaborations.

The use of metaphoric gestures by speakers has long been known to influence thought in the viewer. What is less clear is the extent to which the expression of multiple metaphors in a single gesture reliably affect viewer interpretation. Additionally, gestures which express only one metaphor are not sufficient to explain the broad array of metaphoric gestures and metaphoric scenes that human speakers naturally produce. In this paper we address three issues related to the implementation of metaphoric gestures in virtual humans. First, we break down naturally occurring examples of multiple-metaphor gestures, as well as metaphoric scenes created by gesture sequences. Then, we show the importance of capturing multiple metaphoric aspects of gesture with a behavioral experiment using crowdsourced judgements of videos of alterations of the naturally occurring gestures. Finally, we discuss the challenges for computationally modeling metaphoric gestures that are raised by our findings.

In this paper, we describe the implementation and performance of a Virtual Audience perception model for Virtual Reality (VR). The model is a VR adaptation of an existing desktop model. The system allows a user in VR to easily build and experience a wide variety of atmospheres with small or large groups of virtual agents.The paper describes results of early evaluations for this model in VR. Our first scalability benchmark results demonstrated the ability to simultaneously handle one hundred virtual agents without significantly affecting there commended frame rate for VR applications.This research is conducted in the context of a classroom simulation software for teachers’ training.

In many applications, Embodied Conversational Agents (ECAs) must be able to express various affects such as emotions or social attitudes. Non-verbal signals, such as smiles or gestures, contribute to the expression of attitudes. Social attitudes affect the whole behavior of a person: they are “characteristic of an affective style that colors the entire interaction” [1] . Moreover, recent findings have demonstrated that non-verbal signals are not interpreted in isolation but along with surrounding signals. Non-verbal behavior planning models designed to allow ECAs to express attitudes should thus consider complete sequences of non-verbal signals and not only signals independently of one another. However, existing models do not take this into account, or in a limited manner. The contribution of this paper is a methodology for the automatic extraction of sequences of non-verbal signals characteristic of a social phenomenon from a multimodal corpus, and a non-verbal behavior planning model that takes into account sequences of non-verbal signals rather than signals independently. This methodology is applied to design a virtual recruiter capable of expressing social attitudes, which is then evaluated in and out of an interaction context.

Multimodal interaction technologies have enabled new applications for training interpersonal skills such as public speaking. Various training paradigms have been proposed, most of them relying on some form of graphical feedback provided to the trainee in real-time during their training or after training using an after-action review tool. Another paradigm consists of using virtual characters to provide feedback through their behavior during simulated social interactions. Preliminary studies have started to explore the effectiveness of these different training paradigms; however, these have not investigated the impact of individual differences on which interaction paradigm is more efficient or motivating for different populations of users. In this article, we explore the impact of personality, public speaking anxiety, and immersive tendencies on the experiences of users training public speaking with an interactive virtual audience system providing realtime feedback through virtual audience behavior as well as delayed feedback with an after-action review tool. We found that these three factors impacted different output measures of user experience and user ratings of the system's quality.

Advances in artificial intelligence and in particular machine learning and neural networks have given rise to a new generation of virtual assistants and chatbots. Within this work, we present NADiA - Neurally Animated Dialog Agent - that leverages both the user's verbal input as well as their facial expressions to respond in a meaningful way. NADiA combines a neural language model that generates appropriate responses to user prompts, a convolutional neural network for facial expression analysis, and virtual human technology that is deployed on a mobile phone. Here, we evaluate NADiA's anthropomorphic characteristics and its ability to understand the human interlocutor using both subjective as well as objective measures. We find that NADiA significantly outperforms state of the art chatbot technology and produces comparable behavior to human generated reference outputs.

Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, Stefan Scherer. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

An important aspect of public speaking is delivery, which consists of the appropriate use of non-verbal cues to strengthen the message. Recent works have successfully predicted ratings of public speaking delivery aspects using the entire presentations of speakers. However, in other contexts, such as the assessment of personality or the prediction of job interview outcomes, it has been shown that thin slices, brief excerpts of behavior, provide enough information for raters to make accurate predictions. In this paper, we consider the use of thin slices for predicting ratings of public speaking behavior. We use a publicly available corpus of public speaking presentations and obtain ratings of full videos and thin slices. We first study how thin slices ratings are related to full video ratings. Then, we use automatic audio-visual feature extraction methods and machine learning algorithms to create models for predicting public speaking ratings, and evaluate these models for predicting thin slices ratings and full videos ratings.

It is commonly known that a relationship exists between the human voice and various emotional states. Past studies have demonstrated changes in a number of vocal features, such as fundamental frequency f0 and peakSlope, as a result of varying emotional state. These voice characteristics have been shown to relate to emotional load, vocal tension, and, in particular, stress. Although much research exists in the domain of voice analysis, few studies have assessed the relationship between stress and changes in the voice during a dyadic team interaction. The aim of the present study was to investigate the multimodal interplay between speech and physiology during a high-workload, high-stress team task. Specifically, we studied task-induced effects on participants' vocal signals, specifically, the f0 and peakSlope features, as well as participants' physiology, through cardiovascular measures. Further, we assessed the relationship between physiological states related to stress and changes in the speaker's voice. We recruited participants with the specific goal of working together to diffuse a simulated bomb. Half of our sample participated in an "Ice Breaker" scenario, during which they were allowed to converse and familiarize themselves with their teammate prior to the task, while the other half of the sample served as our "Control". Fundamental frequency (f0), peakSlope, physiological state, and subjective stress were measured during the task. Results indicated that f0 and peakSlope significantly increased from the beginning to the end of each task trial, and were highest in the last trial, which indicates an increase in emotional load and vocal tension. Finally, cardiovascular measures of stress indicated that the vocal and emotional load of speakers towards the end of the task mirrored a physiological state of psychological "threat".

New technological developments in the field of multimodal interaction show great promise for the improvement and assessment of public speaking skills. However, it is unclear how the experience of non-native speakers interacting with such technologies differs from native speakers. In particular, non-native speakers could benefit less from training with multimodal systems compared to native speakers. Additionally, machine learning models trained for the automatic assessment of public speaking ability on data of native speakers might not be performing well for assessing the performance of non-native speakers. In this paper, we investigate two aspects related to the performance and evaluation of multimodal interaction technologies designed for the improvement and assessment of public speaking between a population of English native speakers and a population of non-native English speakers. Firstly, we compare the experiences and training outcomes of these two populations interacting with a virtual audience system designed for training public speaking ability, collecting a dataset of public speaking presentations in the process. Secondly, using this dataset, we build regression models for predicting public speaking performance on both populations and evaluate these models, both on the population they were trained on and on how they generalize to the second population.

Good public speaking skills convey strong and effective communication, which is critical in many professions and used in everyday life. The ability to speak publicly requires a lot of training and practice. Recent technological developments enable new approaches for public speaking training that allow users to practice in a safe and engaging environment. We explore feedback strategies for public speaking training that are based on an interactive virtual audience paradigm. We investigate three study conditions: (1) a non-interactive virtual audience (control condition), (2) direct visual feedback, and (3) nonverbal feedback from an interactive virtual audience. We perform a threefold evaluation based on self-assessment questionnaires, expert assessments, and two objectively annotated measures of eye-contact and avoidance of pause fillers. Our experiments show that the interactive virtual audience brings together the best of both worlds: increased engagement and challenge as well as improved public speaking skills as judged by experts.

We have developed an interactive virtual audience platform for public speaking training. Users' public speaking behavior is automatically analyzed using multimodal sensors, and ultimodal feedback is produced by virtual characters and generic visual widgets depending on the user's behavior. The flexibility of our system allows to compare different interaction mediums (e.g. virtual reality vs normal interaction), social situations (e.g. one-on-one meetings vs large audiences) and trained behaviors (e.g. general public speaking performance vs specific behaviors).

The ability to speak proficiently in public is essential for many professions and in everyday life. Public speaking skills are difficult to master and require extensive training. Recent developments in technology enable new approaches for public speaking training that allow users to practice in engaging and interactive environments. Here, we focus on the automatic assessment of nonverbal behavior and multimodal modeling of public speaking behavior. We automatically identify audiovisual nonverbal behaviors that are correlated to expert judges' opinions of key performance aspects. These automatic assessments enable a virtual audience to provide feedback that is essential for training during a public speaking performance. We utilize multimodal ensemble tree learners to automatically approximate expert judges' evaluations to provide post-hoc performance assessments to the speakers. Our automatic performance evaluation is highly correlated with the experts' opinions with r = 0.745 for the overall performance assessments. We compare multimodal approaches with single modalities and find that the multimodal ensembles consistently outperform single modalities.

This paper presents the analysis of the multimodal behavior of experienced practitioners of job interview coaching, and describes a methodology to specify their behavior in Embodied Conversational Agents acting as virtual recruiters displaying different interpersonal stances. In a first stage, we collect a corpus of videos of job interview enactments, and we detail the coding scheme used to encode multimodal behaviors and contextual information. From the annotations of the practitioners' behaviors we observe specificities of behavior across different levels, namely monomodal behavior variations, inter-modalities behavior influences, and contextual influences on behavior. Finally we propose the adaptation of an existing agent architecture to model these specificities in a virtual recruiter's behavior.