In this study, we developed SCAPED, a platform for running mass experiments with voice-based conversational agents. The main purpose of the toolkit is to study particular aspects of dialogues with an agent. The platform can be integrated into crowdsourcing platforms, such as Prolific as a source for participants and only requires a web browser. It can therefore be used in large-scale experiments. We present a proof-of-concept study which addresses the question of laughter accompanying apologies in a dialogues about the assessments of artworks.

International audience

Motivational Interviewing (MI) is a widely validated approach to behavior change, but existing virtual MI agents operate only in one-on-one settings, ignoring the cost-effectiveness and peer-support dynamics of group MI. We present a simulation environment and reinforcement learning (RL) based dialogue manager for group MI, in which a discrete Soft-Actor-Critic (SAC) policy selects therapist dialogue acts and a large language model generates utterances, with two LLM-prompted patient agents as interlocutors. Our model supports adaptation to different participant profiles. We compared our dialogue manager with four LLM-based ones at the dialogue acts level. We observed that RL yields a significantly different therapist policy, which showed the tendency to generate more directive acts and adapt to varying group compositions. Participant profile adaptation was the strongest in groups containing an open-to-change participant.

Maintaining mutual understanding is a key component in human-human conversation to avoid conversation breakdowns, in which repair, particularly Other-Initiated Repair (OIR, when one speaker signals trouble and prompts the other to resolve), plays a vital role.However, Conversational Agents (CAs) still fail to recognize user repair initiation, leading to breakdowns or disengagement.This work proposes a multimodal model to automatically detect repair initiation in Dutch dialogues by integrating linguistic and prosodic features grounded in Conversation Analysis.The results show that prosodic cues complement linguistic features and significantly improve the results of pretrained text and audio embeddings, offering insights into how different features interact.Future directions include incorporating visual cues, exploring multilingual and cross-context corpora to assess the robustness and generalizability.

As the quest for high-level realism in virtual environments continues, there is a growing demand for accurate representations of human characteristics in Virtual Humans (VHs), including gender and emotions.Although people assign gender to VHs even without explicit cues, it is still unknown whether this bias endures during interactive engagement rather than passive video observation.This study aims to address this issue by conducting a perceptual study involving the evaluation of a genderless Virtual Baby (VB) in an interactive web environment.This study advances the understanding of gender assignment in interactive environments.

International audience

Figure 1: Greta 2.0 system architecture: Green color indicates added modules to the Greta platform, blue indicates existing modules of the Greta platform following the SAIBA framework, gray indicates abstract concepts, and purple indicates types of XML files that exchange information between modules.Each new module in green is explained and described in the paper.

Current computational models for humour recognition and laughter generation in dialogue systems face significant limitations in explainability, context consideration and adaptability. This paper approaches these challenges by investigating how humour recognition develops in its earliest forms—during the first year of life. Drawing on developmental psychology and cognitive science, we propose a formal model incorporated within the KoS dialogue framework. This model captures how infants evaluate potential humour through knowledge-based appraisal and context-dependent modulation, including safety, emotional state, and social cues. Our model formalises dynamic knowledge updates during the dyadic interaction. We believe that this formal model can serve as the basis for developing more natural humour appreciation capabilities in dialogue systems and can be implemented in a robotic platform.

This paper addresses the challenge of transferring the behaviour expressivity style of a virtual agent to another one while preserving behaviour shape as they carry communicative meaning. Behaviour expressivity style is viewed here as the qualitative properties of behaviours. We propose TranSTYLer , a multimodal transformer-based model that synthesises the multimodal behaviours of a source speaker with the style of a target speaker. We assume that behaviour expressivity style is encoded across various modalities of communication, including text, speech, body gestures, and facial expressions. The model employs a style-content disentanglement schema to ensure that the transferred style does not interfere with the meaning conveyed by the source’s behaviours. Our approach eliminates the need for style labels and allows the generalisation of styles not seen during the training phase. We train our model on the PATS corpus , which we extended to include dialogue acts and 2D facial landmarks. Objective and subjective evaluations show that our model outperforms state-of-the-art models in style transfer for both seen and unseen styles during training. To tackle the issues of style and content leakage that may arise, we propose a methodology to assess the degree to which behaviour and gestures associated with the target style are successfully transferred while ensuring the preservation of the ones related to the source content.

In this paper, we present a decision model aimed at determining when a social touch by a socially interactive agent (SIA) would be considered socially ‘correct’ : both coherent (meaningful given the current situation) and acceptable to the human interlocutor. Those decisions are computed by taking the context of interaction and the estimated level of rapport between the human and the agent into account. We then present a study based on a vicarious protocol where participants evaluated recorded interactions between an avatar (a human-controlled 3D character) and our agent. Results indicate that the estimations made by our decision model regarding the state of the situation mirror those made by human observers. Our initial hypothesis that the level of rapport would positively influence the acceptability of a touch receives mixed support. Social touch occurrences did not feel as coherent and acceptable as hoped yet. Avenues of improvement for human-agent interactions including social touch are discussed.

In video games, non-player characters (NPCs) have an essential role in shaping players' experiences. The design of their appearance and their behaviors can be manipulated in coherence and consistency to maintain players' expectations or on the contrary to induce surprise. The influence of NPCs’ coherence and consistency on players’ evaluation of them remains to be unveiled. To fill this gap, two experiments were conducted in the context of a military shooter game. Players’ evaluation of NPCs’ perceived intelligence and believability, were measured, as these two dimensions are fundamental for their adoption and engagement toward them. The first experiment investigated the impact of breaking players' initial expectations on the evaluation of NPCs. The second experiment focused on the influence of NPCs’ coherence and consistency on both players' expectations and evaluation of NPCs, by means of a combination of questionnaires, behavioral, and physiological measures. Our results reveal that breaking players' expectations influence their evaluation of NPCs, with coherent and consistent design reinforcing expectations and incoherent design challenging them.

Motivational interviewing (MI) is a client-centered counseling style that addresses (the client) user’s motivation for behavior change. In this paper, we present a behavior generation model for Socially Interactive Agents (SIA) and apply it to an SIA acting as a virtual therapist in (MI). MI defines different types of dialogue acts for therapist and client. It has been shown that therapist builds rapport with their client by adapting their verbal and nonverbal behaviors. Based on the analysis of a human-human MI dataset (AnnoMI), we found co-occurrences between facial expressions and dialogue acts for both therapist and client. Moreover, the therapist adapts their behavior to their client’s behavior to favor rapport. Our behavior generation model embeds these co-occurrences as well as such behavior adaptation. To this aim, we build an observation-to-action framework based on a conditional diffusion approach trained on the AnnoMI corpus. Our model learns to generate the virtual therapist’s facial expressions conditioned by MI dialogue acts and the client’s nonverbal behaviors. We aim to make SIAs more effective in therapy-like interactions, by using user’s behaviors in addition to contextual information (i.e. dialogue acts and nonverbal behaviors of both user and agent) to drive the SIA behavior.

Authors : Nezih Younsi, Catherine Pelachaud, Laurence Chaby Title : Beyond Words: Decoding Facial Expression Dynamics in Motivational Interviewing Abstract : This paper focuses on studying the facial expressions of both client and therapist in the context of Motivational Interviewing (MI). The annotation system Motivational Interview Skill Code MISC defines three types of talk, namely sustain, change, and neutral for the client and information, question, or reflection for the therapist. Most studies on MI look at the verbal modality. Our research aims to understand the variation and dynamics of facial expressions of both interlocutors over a counseling session. We apply a sequence mining algorithm to identify categories of facial expressions for each type. Using co-occurrence analysis, we derive the correlation between the facial expressions and the different types of talk, as well as the interplay between interlocutors’ expressions.

Motivational Interviewing (MI) is an approach to therapy that emphasizes collaboration and encourages behavioral change. To evaluate the quality of an MI conversation, client utterances can be classified using the MISC code as either Change Talk (CT), Sustain Talk (ST), or Follow/Neutral (F/N). The proportion of CT in an MI conversation positively correlates with therapy outcomes, making accurate classification of client utterances essential. This paper presents a classifier that accurately distinguishes between the three MISC classes (CT, ST, and F/N), leveraging multimodal features such as text, prosody, and facial expressivity. We annotate the publicly available AnnoMI dataset to train our model to collect multimodal information. Furthermore, we identify the modality that contributes most to the decision-making process, providing valuable insights into the interplay of different modalities during an MI conversation.

Over the years, with my group, we have conducted different studies to endow expressive communicative capabilities to socially interactive agents. We first focused on developing models for generating a wide variety of emotional expressions and social attitudes. While our initial research was geared toward the agent only, we then turned our attention to modeling agents as active interlocutors of human users. To such an aim, we integrated different adaptation mechanisms in the human-agent platform Greta. We implemented phenomena such as imitation, intra- and inter-synchronization, and conversational strategies. We conducted evaluation studies to measure the impact on users’ perception of the agent and of the interaction quality. In this talk, I will present some of our earlier works as well as our most recent ones.

The demand for mental health services has outpaced available resources, resulting in long wait times for patients. A potential solution is to use virtual agents that perform motivational interviews. These agents can be rule-based, requiring expert knowledge, or data-driven, needing large datasets for training, which are often hard to obtain. Patient simulation can generate synthetic data as an alternative. Traditionally, this involved template utterances with a dialog manager or uncontrollable black box large language models LLMs. This study proposes a hybrid approach, leveraging both methods to see if LLMs can follow instructed dialog acts while generating natural, coherent utterances. Our study shows that the language model adheres to given conditions and that conditioning on dialog improves the naturalness and coherence of generated utterances, validating our approach for simulating patient responses.

Co-speech gestures are fundamental for communication. The advent of recent deep learning techniques has facilitated the creation of lifelike, synchronous co-speech gestures for Embodied Conversational Agents. "In-the-wild" datasets, aggregating video content from platforms like YouTube via human pose detection technologies, provide a feasible solution by offering 2D skeletal sequences aligned with speech. Concurrent developments in lifting models enable the conversion of these 2D sequences into 3D gesture databases. However, it is important to note that the 3D poses estimated from the 2D extracted poses are, in essence, approximations of the ground-truth, which remains in the 2D domain. This distinction raises questions about the impact of gesture representation dimensionality on the quality of generated motions. Our study examines the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models.

The demand for mental health services has risen substantially in recent years, leading to challenges in meeting patient needs promptly.Virtual agents capable of emulating motivational interviews (MI) have emerged as a potential solution to address this issue, offering immediate support that is especially beneficial for therapy modalities requiring multiple sessions.However, developing effective patient simulation methods for training MI dialog systems poses challenges, particularly in generating syntactically and contextually correct, and diversified dialog acts while respecting existing patterns and trends in therapy data.This paper investigates data-driven approaches to simulate patients for training MI dialog systems.We propose a novel method that leverages time series models to generate diverse and contextually appropriate patient dialog acts, which are then transformed into utterances by a conditioned large language model.Additionally, we introduce evaluation measures tailored to assess the quality and coherence of simulated patient dialog.Our findings highlight the effectiveness of dialog act-conditioned approaches in improving patient simulation for MI, offering insights for developing virtual agents to support mental health therapy.

In daily conversations, people often encounter problems prompting conversational repair to enhance mutual understanding.By employing an automatic coreference solver, alongside examining repetition, we identify various linguistic features that distinguish turns when the addressee initiates repair from those when they do not.Our findings reveal distinct patterns that characterize the repair sequence and each type of other-repair initiation.

When we converse, we adapt our behaviors to our interlocutors. The adaptation can serve to indicate our engagement which can also elicit enhancement of the involvement of others. Virtual agents (or socially interactive virtual agents) that play the role of interaction partners can improve the human users’ interaction experience by displaying continuous and adaptive behaviors in real time. Virtual agents have been used in multiple domains to improve user interaction and performance. The promising results of the endowment of adaptation to agents in increasing the agents’ perception and user experience were shown in previous studies. In this paper, we develop an adaptive virtual agent that renders real-time adaptive behaviors based on the behaviors shown by its human interlocutor. The ASAP model rendering reciprocally adaptive agent behavior was employed to realize the system. The system consists of four main parts: perception of social signals, agent adaptive behavior generation, agent visualization (i.e. rendering of the agent’s verbal and nonverbal behavior), and communication of signals. To showcase the usefulness of our adaptive agent, as a proof-of-concept we choose the e-health application of cognitive behavior therapy (CBT), which identifies and rectifies biased and irrational thoughts (or automatic thoughts). Through this study, we show the importance of giving the agent reciprocal adaptation capability notably in enhancing the user experience and the effectiveness of the CBT session. We validate the importance of endowing such adaptation capability by studying the difference between agents that are reciprocal adaptive, solely expressive (with mismatched behavior), and inexpressive (in a still posture) via questionnaires and measures related to the agent perception (naturalness, human-likeliness, synchrony, and engagement) for user experience and the CBT effectiveness (mood, anxiety, stress, and cognitive change). These results highlight the value of making virtual agents adapt in real time. This could lead to agents being capable of providing more personalized and interactive experiences for a wide range of applications. Also, we have collected a new human-agent interaction (HAI) database, HAI-CBT database, which is publicly available to the research community.

Non-Playable Characters (NPCs) are a subtype of virtual agents that populate video games by endorsing social roles in the narrative. To infer NPCs’ roles, players evaluate NPCs’ appearance and behaviors, usually by ascribing human traits to NPCs, such as intelligence, likability and morality. In particular, hostile NPCs in video games are essential to build the games’ inherent challenges. The three experiments reported here investigated the extent to which the perception of hostility in a military shooter game (including both threat of appearance and aggressiveness in behaviors) is influenced by the appearance and the behaviors of NPCs thanks to perceived intelligence, likability and morality-related questionnaires. Our results first show that hostility is efficiently conveyed through NPCs’ behaviors, but not significantly by their appearance. Second, our study allows identifying the main predictors of hostility perception, namely unfriendliness, knowledge and harmfulness.

We propose a multimodal speech driven approach to generate 2D upper-body gestures for virtual agents, in the communicative style of different speakers, seen or unseen by our model during training. Upper-body gestures of a source speaker are generated based on the content of his/her multimodal data - speech acoustics and text semantics. The synthesized source speaker's gestures are conditioned on the multimodal style representation of the target speaker. Our approach is zero-shot, and can generalize the style transfer to new unseen speakers, without any additional training. An objective evaluation is conducted to validate our approach.

During an interaction, people exchange speaking turns by coordinating with their partners. Exchanges can be done smoothly, with pauses between turns or through interruptions. Previous studies have analyzed various modalities to investigate turn shifts and their types (smooth turn exchange, overlap, and interruption). Modality analyses were also done to study the interpersonal synchronization which is observed throughout the whole interaction. Likewise, we intend to analyze different modalities to find a relationship between the different turn switch types and interpersonal synchrony. In this study, we provide an analysis of multimodal features, focusing on prosodic features (F0 and loudness), head activity, and facial action units, to characterize different switch types.

Endowing socially interactive virtual agents with the social touch modality could improve their emotional communication abilities and help them bond with human users. Touch is however a sensitive channel of communication that can have negative effects if used inappropriately. We present a first implementation of a system enabling an agent to perform social touch-based interactions and a preliminary study of the system dedicated to determining the factors that come into play for the social acceptability of an agent-initiated touch. The relationship between social touch with a virtual agent and sense of embodiment is also discussed. Based on the results of the study, we propose social (macro) and practical (micro) insights into how to produce meaningful, coherent and acceptable touch decisions (when to touch or not depending on the situation).

Gaze and laughter play a crucial role in managing miscommunication and coordinating social interactions. We hypothesise that models of laughter and gaze coordination in human dialogue extend to virtual entities. This paper describes methodology of the future experiment which involves a socially interactive agent (SIA) that incorporates previous theoretical findings.

International audience

Socially Interactive Agents (SIAs) offer users with interactive face-to-face conversations. They can take the role of a speaker and communicate verbally and nonverbally their intentions and emotional states; but they should also act as active listener and be an interactive partner. In human-human interaction, interlocutors adapt their behaviors reciprocally and dynamically. The endowment of such adaptation capability can allow SIAs to show social and engaging behaviors. In this paper, we focus on modelizing the reciprocal adaptation to generate SIA behaviors for both conversational roles of speaker and listener. We propose the Augmented Self-Attention Pruning (ASAP) neural network model. ASAP incorporates recurrent neural network, attention mechanism of transformers, and pruning technique to learn the reciprocal adaptation via multimodal social signals. We evaluate our work objectively, via several metrics, and subjectively, through a user perception study where the SIA behaviors generated by ASAP is compared with those of other state-of-the-art models. Our results demonstrate that ASAP significantly outperforms the state-of-the-art models and thus shows the importance of reciprocal adaptation modeling.

Conversational agents (CAs) in narrative experiences are defined by the role they endorse and the communication style they adopt when users interact with them. In computer games, users' perception of intelligence and believability ascription influence the positive evaluation of CAs. Yet, the impact of CAs' role and communication style on users' experience remains to be clarified. In this research, the effect of the role and communication style of CAs on users' evaluation is investigated in a crime-solving textual game. Different CAs were created whose roles in the narrative (witness or suspect) and communication style (aggressive or cooperative) were manipulated. A Wizard of Oz method was used to control communication style while users' experience was assessed regarding their interaction with each CA using scales of perceived intelligence and believability. Users also had to indicate a culprit and rate the certainty of their judgments. The results show that both CAs' role and communication style have an influence on users' perception of intelligence and believability, with a higher effect of the role. However, only communication style had a significant influence on the choice of the culprit.

During an interaction, partners adapt their behaviors to each other. Adaptation can have several functions such as being a sign of engagement and enhancing human users' interaction experience. It is important that virtual agents acting as interaction partners should continuously adapt their behaviors to those of their interlocutors in real time. This paper focuses on creating an interactive virtual agent that is capable of rendering real-time adaptive behaviors in response to its human interlocutor. It ensures the two aspects: generating real-time adaptive behavior and managing natural dialogue. We propose a system of an adaptive virtual agent and choose the e-health application of Cognitive Behavioral Therapy (CBT), which is a mental health treatment that restructures automatic thoughts into balanced thoughts, as a proof-of-concept to showcase the benefit of endowing behavior adaptation to the agent. The virtual agent adapts to the user via the display of nonverbal behaviors, which are generated via a deep learning model, throughout the whole interaction while acting as a therapist helping human users to detect their negative automatic thoughts.

When conversing, people adapt their behaviors to one another to show their engagement. Virtual agents, acting as interaction partners, should also adapt to their interlocutors in real time. In this paper, we introduce a virtual agent delivering Cognitive Behavioral Therapy (CBT) and adapting its behaviors in real time. The system focuses on the real-time generation of adaptive behavior and management of natural CBT dialogue.

Interruptions are an important aspect of human-human communication. They help to adjust the conversation flow. Our aim is to equip virtual agents with the ability to handle interruptions, that is to decide when and how to interrupt their human interlocutor. In this paper, we focus on predicting when interruptions may occur during the conversation using multimodal features only from the speaker and propose a model trained on a corpus of dyadic interactions. To assess the model's accuracy, we conduct a perceptual study where we compare different timings (ground truth, randomly chosen or predicted by our model).

Virtual humans and social robots frequently generate behaviors that human observers naturally see as expressing emotion. In this review article, we highlight that these expressions can have important benefits for human–machine interaction. We first summarize the psychological findings on how emotional expressions achieve important social functions in human relationships and highlight that artificial emotional expressions can serve analogous functions in human–machine interaction. We then review computational methods for determining what expressions make sense to generate within the context of interaction and how to realize those expressions across multiple modalities, such as facial expressions, voice, language, and touch. The use of synthetic expressions raises a number of ethical concerns, and we conclude with a discussion of principles to achieve the benefits of machine emotion in ethical ways.

International audience

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user’s behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

During an interaction interruptions occur frequently. Interruptions may arise to fulfill different goals such as changing the topic of conversation abruptly, asking for clarification, completing the current speaker’s turn. Interruptions may be cooperative or competitive depending on the interrupter’s intention. Our main goal is to endow a Socially Interactive Agent with the capacity to handle user interruptions in dyadic interaction. It requires the agent to detect an interruption and recognize its type (cooperative/competitive), and then to plan its behaviours to respond appropriately. As a first step towards this goal, we developed a multimodal classification model using acoustic features, facial expression, head movement, and gaze direction from both, the interrupter and the interruptee. The classification model learns from the sequential information to automatically identify interruptions type. We also present studies we conducted to measure the shortest delay needed (0.6s) for our classification model to identify interruption types with a high classification accuracy (81%). On average, most interruption overlaps last longer than 0.6s, so a Socially Interactive Agent has time to detect and recognize an interruption type and can respond in a timely manner to its human interlocutor’s interruption.

In this work, we present a socially interactive agent able to adapt its conversational strategies to maximize user's engagement during the interaction. For this purpose, we train our agent with simulated users using deep reinforcement learning. First, the agent estimates the simulated user's engagement depending on the latter's nonverbal behaviors and turn-taking status. This measured engagement is then used as a reward to balance the task of the agent (giving information) and its social goal (maintaining the user highly engaged). Agent's dialog acts may have different impact on the user's engagement depending on the latter's conversational preferences.

Integrating the existing interruption and turn switch classification methods, we propose a new annotation schema to annotate different types of interruptions through timeliness, switch accomplishment and speech content level. The proposed method is able to distinguish smooth turn exchange, backchannel and interruption (including interruption types) and to annotate dyadic conversation. We annotated the French part of NoXi corpus with the proposed structure and use these annotations to study the probability distribution and duration of each turn switch type.

In social interactions between humans and Embodied Conversational Agents (ECAs) conversational interruptions may occur. ECAs should be prepared to detect, manage and react to such interruptions in order to keep the interaction smooth, natural and believable. In this paper, we examined nonverbal reactions exhibited by an interruptee during conversational interruptions and we propose a novel technique driven by an evolutionary algorithm to build a computational model for ECAs to manage user's interruptions. We propose a taxonomy of conversational interruptions adapted from social psychology, an annotation schema for semi-automatic detection of user's interruptions and a corpus-based observational analysis of human nonverbal reactions to interruptions. Then we present a methodology for building an ECA behavioral model including the design and realization of an interactive study driven by an evolutionary algorithm, where participants interactively built the most appropriate set of multimodal reactive behaviours for an ECA to display interpersonal attitudes (friendly/hostile) through nonverbal reactions to a conversational interruption.

An Embodied Conversational Agent (ECA) is a virtual character designed to interact with humans in the most natural way. In the recent years, ECAs have been deployed in various contexts, such as commercial consulting and social training. In the context of social training, the virtual agent should be able to express different social attitudes in order to train the user in different situations, likely to occur in real life. Previous studies from psychology underlined the importance of considering the non-verbal behavior as well as its evolution over time, for efficient modeling of interpersonal attitudes. Inspired by these works as well as by advances from sequence mining, we propose to model attitude variation as a sequence of non-verbal signals, each being described by its starting time and duration. We demonstrate the efficiency of our model by integrating the sequences representing attitude variation in an ECA and assessing the obtained results based on the interpersonal circumplex, statistical tests and accuracy measures. To the best of our knowledge, this is the first attempt to study the relationship, in term of perception, between different attitude variations.

Humans have the ability to convey an array of emotions through complex and rich touch gestures. However, it is not clear how these touch gestures can be reproduced through interactive systems and devices in a remote mediated communication context. In this article, we explore the design space of device-initiated touch for conveying emotions with an interactive system reproducing a collection of human touch characteristics. For this purpose, we control a robotic arm to touch the forearm of participants with different force, velocity and amplitude characteristics to simulate human touch. In view of adding touch as an emotional modality in human-machine interaction, we have conducted two studies. After designing the touch device, we explore touch in a context-free setup and then in a controlled context defined by textual scenarios and emotional facial expressions of a virtual agent. Our results suggest that certain combinations of touch characteristics are associated with the perception of different degrees of valence and of arousal. Moreover, in the case of non-congruent mixed signals (touch, facial expression, textual scenario) not conveying a priori the same emotion, the message conveyed by touch seems to prevail over the ones displayed by the visual and textual signals.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user’s behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

The development of applications with intelligent virtual agents (IVA) often comes with integration of multiple complex components. In this article we present the Agents United Platform: an open source platform that researchers and developers can use as a starting point to setup their own multi-IVA applications.

International audience

Group conversation is a frequently used form of communication for exchanging ideas and making decisions. Cohesion is an emergent phenomenon that describes the members' attraction towards the group and towards working together. In this paper, we present the cohesion labels assigned to segments from [redacted], a multimodal dataset of simulated medical consultations. Then, we present the analysis performed to identify social cues that characterize cohesion and report the accuracy for classifying cohesion. Results show that non-verbal social cues like gaze, facial AUs, laughter etc., indeed convey information regarding the level of cohesion. Finally we present a preliminary evaluation conducted using the prominent cues to simulate a cohesive group of agents.

Turn management is one of the necessary social interactions skills. In human-human interactions, turn changes are naturally completed by interruption, "cooperatively" or "competitively". Interruptions are inherent in conversation. They can be considered disruptive at first glance, but can also be cooperative and participate to enriching the interaction. To create natural human-agent interaction, Embodied Conversational Agent (ECA) should be able to communicate autonomously with humans both verbally and nonverbally. A challenge is then to handle interruptions during their interaction. This article presents our ongoing work to endow ECA to manage interruption during the interaction with a human partner. In order to achieve this goal, we start by analyzing human-human interaction data.

International audience

"Fillers", example "um" in English, have been linked to the "Feeling of Another’s Knowing (FOAK)" or the listener’s perception of a speaker’s expressed confidence. Yet, in Spoken Language Processing (SLP) they remain unexplored, or overlooked as noise. We introduce a new and challenging task, that is the prediction of FOAK, which we think has widespread applicability, given the increasing popularity of automatic processing of educational and job interviews, reviews and speeches. We design a set of filler features based on linguistic literature, and investigate their potential in FOAK prediction. We show that the integration of information related to implicature meanings allows an improvement in the FOAK model and that the different functions of fillers are differently correlated with confidence.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user's behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

Politeness behaviors could affect individuals' decisions heavily in their daily lives and may therefore also play an important role in human-agent interactions. This study considers the impact of politeness behaviors made by a virtual agent, already in a small face-to-face conversational group with another agent, on a human participant as they approach to join it in a virtual environment displayed on a monitor. The agent uses five verbal and nonverbal politeness strategies, ranging from indirect and implicit to direct and explicit, in an attempt to influence the participant to join the group at an inconvenient location, which requires more time and effort than a direct route that would ignore the invitation of the agent. In addition to assessing the success of the strategies at influencing participant behavior, the participants' perception of the agent's persuasive behavior is assessed in relation to clarity, face loss, positive face, and negative face. Based on results from a within-subjects experiment with 30 participants, we found that more direct and explicit politeness strategies have a higher level of success when requesting a participant to join a small group at an inconvenient location, but sometimes negatively impact their perception of the agent. A positive politeness strategy was found to be the most effective for both persuasive success and maintaining a positive impression of the agent.

ISO standard 24617-2 for dialogue act annotation, established in 2012, has in the past few years been used both in corpus annotation and in the design of components for spoken and multimodal dialogue systems. This has brought some inaccuracies and undesirbale limitations of the standard to light, which are addressed in a proposed second edition. This second edition allows a more accurate annotation of dependence relations and rhetorical relations in dialogue. Following the ISO 24617-4 principles of semantic annotation, and borrowing ideas from EmotionML, a triple-layered plug-in mechanism is introduced which allows dialogue act descriptions to be enriched with information about their semantic content, about accompanying emotions, and other information, and allows the annotation scheme to be customised by adding application-specific dialogue act types.

Group cohesion is an emergent phenomenon that describes the tendency of the group members’ shared commitment to group tasks and the interpersonal attraction among them. This paper presents a multimodal analysis of group cohesion using a corpus of multi-party interactions. We utilize 16 two-minute segments annotated with cohesion from the AMI corpus. We define three layers of modalities: non-verbal social cues, dialogue acts and interruptions. The initial analysis is performed at the individual level and later, we combine the different modalities to observe their impact on perceived level of cohesion. Results indicate that occurrence of laughter and interruption are higher in high cohesive segments. We also observe that, dialogue acts and head nods did not have an impact on the level of cohesion by itself. However, when combined there was an impact on the perceived level of cohesion. Overall, the analysis shows that multimodal cues are crucial for accurate analysis of group cohesion.

This paper presents a computational model for managing an Embodied Conversational Agent's first impressions of warmth and competence towards the user. These impressions are important to manage because they can impact users' perception of the agent and their willingness to continue the interaction with the agent. The model aims at detecting user's impression of the agent and producing appropriate agent's verbal and nonverbal behaviours in order to maintain a positive impression of warmth and competence. User's impressions are recognized using a machine learning approach with facial expressions (action units) which are important indicators of users' affective states and intentions. The agent adapts in real-time its verbal and nonverbal behaviour, with a reinforcement learning algorithm that takes user's impressions as reward to select the most appropriate combination of verbal and non-verbal behaviour to perform. A user study to test the model in a contextualized interaction with users is also presented. Our hypotheses are that users' ratings differs when the agents adapts its behaviour according to our reinforcement learning algorithm, compared to when the agent does not adapt its behaviour to user's reactions (i.e., when it randomly selects its behaviours). The study shows a general tendency for the agent to perform better when using our model than in the random condition. Significant results shows that user's ratings about agent's warmth are influenced by their a-priori about virtual characters, as well as that users' judged the agent as more competent when it adapted its behaviour compared to random condition.

The representation of expressive body movement is a critical step to automatically classify bodily expression of emotions. Using motion capture data, we study the contribution of different types of body features to the classification of emotions. In particular, we focus on the role played by temporal profiles of motion cues with regards to a set of multi-level body cues, including postural and dynamic ones.

A social interaction implies a social exchange between two or more persons, where they adapt and adjust their behaviors in response to their interaction partners. With the growing interest in human-agent interactions, it is desirable to make these interactions more natural and human like. In this context, we aim at enhancing the quality of the interaction between user and Embodied Conversational Agent (ECA) by endowing ECA with the capacity to adapt its behavior in real time according the user’s behavior. The novelty of our approach is to model the agent’s nonverbal behaviors as a function of both agent’s and user’s behaviors jointly with the agent’s communicative intentions creating a dynamic loop between both interactants. Moreover, we encompass the variation of behavior over time through a LSTM-based model. Our model IL-LSTM (Interaction Loop LSTM) predicts the next agent’s behavior taking into account the behavior that both, the agent and the user, have displayed within a time window. We have conducted an evaluation study involving an agent interacting with visitors in a science museum. Results of our study show that participants have better experience and are more engaged in the interaction when the agent adapts its behaviors to theirs, thus creating an interactive loop.

In the recent years, engagement modeling has gained increasing attention due the important role it plays in human-agent interaction. The agent should be able to detect, in real time, the engagement level of the user in order to react accordingly. In this context, our goal is to develop a computational model to predict engagement level of the user in real time. Relying on previous findings, we use facial expressions, head movements and gaze direction as predictive features. Moreover, engagement is not only measured from single cues, but from the combination of several cues that arise over a certain time window. Thus, for better engagement prediction, we consider the variation of multimodal behaviors over time. To this end, we rely on LSTM that can jointly model the temporality and the sequentiality of multimodal behaviors.

This paper presents progress and challenges in developing a platform for multi-character, argumentation based, interaction with a group of virtual coaches for healthcare advice and promotion of healthy behaviours. Several challenges arise in the development of such a platform, e.g., choosing the most effective way of utilising argumentation between the coaches with multiple perspectives, handling the presentation of these perspectives and finally, the personalisation and adaptation of the platform to the user types. In this paper, we present the three main challenges recognized, and show how we aim to address these.

Agents (virtual/physical) in a learning environment can be introduced in different roles, such as a tutor, mentor, motivator, expert, peer student etc. Each agent type brings an expertise, creating a unique social relationship with students. Depending on their role, agents have specific goals and beliefs, as well as attitudes towards the learners, thereby influencing different aspects of learning such as cognitive, affective and meta-cognitive processes in a learner. The proposed research will primarily investigate the meta-cognitive aspect of self-regulation in collaborative learning interactions and its variations with various scaffolding strategies based on agent roles. The learning interaction will be based on the socially shared regulation model of self regulation, which accommodates the social context of self regulated learning created by agents in multiples roles and behaviours. The objectives of this research will be to understand how various roles and behaviours of the agents would influence the self regulation skills of the learner and to design a role-based strategy selection model for regulation scaffolding, based on the behavioural, motivational and cognitive measures of the learning interaction.

When interacting with others, we form an impression that can be declined along the two psychological dimensions of warmth and competence. By managing them, high level of engagement in an interaction can be maintained and reinforced. Our aim is to develop a virtual agent that can form and maintain a positive impression on the user that can help in improving the quality of the interaction and the user's experience. In this paper, we present an interactive system in which a virtual agent adopts a dynamic communication strategy during the interaction with a user, aiming at forming and maintaining a positive impression of warmth and competence. The agent continuously analyzes user's non-verbal signals to determine user's engagement level and adapts its communication strategy accordingly. We present a study in which we manipulate the communication strategy of the agent and we measure user's experience and user's perception of the agent's warmth and competence.

Our objective is to develop a machine-learning model that allows a virtual agent to automatically perform appropriate communicative gestures. Our first step is to compute when a gesture should be performed. We express this as classification problem. We initially split the data into NoGesture class and HasGesture class. We develop a model based on recurrent neural network with attention mechanism to compute the class based on the speech prosody. We apply the model on a dialog corpus segmented into different gesture classes and gesture phases. We treat the prosody as the input sequence and the gesture classes as the output sequence.

We propose a paradigm called Skin-On interfaces, in which interactive devices have their own (artificial) skin, thus enabling new forms of input gestures for end-users (e.g. twist, scratch). Our work explores the design space of Skin-On interfaces by following a bio-driven approach: (1) From a sensory point of view, we study how to reproduce the look and feel of the human skin through three user studies;(2) From a gestural point of view, we explore how gestures naturally performed on skin can be transposed to Skin-On interfaces; (3) From a technical point of view, we explore and discuss different ways of fabricating interfaces that mimic human skin sensitivity and can recognize the gestures observed in the previous study; (4) We assemble the insights of our three exploratory facets to implement a series of Skin-On interfaces and we also contribute by providing a toolkit that enables easy reproduction and fabrication.

In many applications, Embodied Conversational Agents (ECAs) must be able to express various affects such as emotions or social attitudes. Non-verbal signals, such as smiles or gestures, contribute to the expression of attitudes. Social attitudes affect the whole behavior of a person: they are “characteristic of an affective style that colors the entire interaction” [1] . Moreover, recent findings have demonstrated that non-verbal signals are not interpreted in isolation but along with surrounding signals. Non-verbal behavior planning models designed to allow ECAs to express attitudes should thus consider complete sequences of non-verbal signals and not only signals independently of one another. However, existing models do not take this into account, or in a limited manner. The contribution of this paper is a methodology for the automatic extraction of sequences of non-verbal signals characteristic of a social phenomenon from a multimodal corpus, and a non-verbal behavior planning model that takes into account sequences of non-verbal signals rather than signals independently. This methodology is applied to design a virtual recruiter capable of expressing social attitudes, which is then evaluated in and out of an interaction context.

A modern way of life needs a modern way of coaching. Despite the proliferation of ICT solutions for personalized health care, there is still no easy way to provide older adults with integrated coaching services. In this paper we introduce the concept of Council of Coaches — a radically new virtual coaching concept based on multiple autonomous, embodied virtual coaches, which form together a personal council that fulfills the needs of older adults in an integrated way. In this concept, coaching takes the form of an open dialog in which clients co-construct together with a selected number of coaches their own plans to go for a healthier lifestyle. Virtual coaches are presented to users by means of embodied conversational social characters. We discuss technical and social challenges on the path towards realizing the Council of Coaches concept, a radically new view of health coaching that involves the state of the art in human-computer interaction, natural dialogue, and argumentation technology.

Virtual humans need to be persuasive in order to promote behaviour change in human users. While several studies have focused on understanding the numerous aspects that influence the degree of persuasion, most of them are limited to dyadic interactions. In this paper, we present an evaluation study focused on understanding the effects of multiple agents on user's persuasion. Along with gender and status (authoritative & peer), we also look at type of focus employed by the agent i. e., user-directed where the agent aims to persuade by addressing the user directly and vicarious where the agent aims to persuade the user, who is an observer, indirectly by engaging another agent in the discussion. Participants were randomly assigned to one of the 12 conditions and presented with a persuasive message by one or several virtual agents. A questionnaire was used to measure perceived interpersonal attitude, credibility and persuasion. Results indicate that credibility positively affects persuasion. In general, multiple agent setting, irrespective of the focus, was more persuasive than single agent setting. Although, participants favored user-directed setting and reported it to be persuasive and had an increased level of trust in the agents, the actual change in persuasion score reflects that vicarious setting was the most effective in inducing behaviour change. In addition to this, the study also revealed that authoritative agents were the most persuasive.

In this paper, we explore the interaction space of MobiLimb, a small 5-DOF serial robotic manipulator attached to a mobile device. It (1) overcomes some limitations of mobile devices (static, passive, motionless); (2) preserves their form factor and I/O capabilities; (3) can be easily attached to or removed from the device; (4) offers additional I/O capabilities such as physical deformation and (5) can support various modular elements such as sensors, lights or shells. We illustrate its potential through three classes of applications: As a tool, MobiLimb offers tangible affordances and an expressive controller that can be manipulated to control virtual and physical objects. As a partner, it reacts expressively to users' actions to foster curiosity and engagement or assist users. As a medium, it provides rich haptic feedback such as strokes, pat and other tactile stimuli on the hand or the wrist to convey emotions during mediated multimodal communications.

In this paper, we examine the perception of emotions as well as the characterization and the classification of emotional body expressions based on perceptual body cues ratings. Emilya (EMotional body expression In daILY Actions), a database of body expressions of eight emotions (including Neutral) in seven daily actions performed by 11 actors, is used for these purposes. A perceptual study is conducted to explore four issues: 1) how expressed emotions are perceived by humans, 2) how emotion recognition by humans differs across daily actions, 3) how expressed emotions are characterized by humans through body cues, and 4) how emotions are automatically classified based on human rating of body cues. Across all the actions, most of the expressed emotions were correctly identified, but some were confused (e.g., Shame and Sadness). Confusions occurring at the level of emotion perception may be due to a lack of contextual factors (Emilya contains body movement of daily actions without reference to a context), to a similarity of bodily expressions, but also to the lack of other modalities that may contribute to a better recognition of bodily expression of these emotions (e.g., facial expressions). In the paper, we detail and discuss the results from these different studies.

In this paper we present an analysis from a corpus of dyadic expert-novice knowledge sharing interactions. The analysis aims at investigating the relationship between observed non-verbal cues and first impressions formation of warmth and competence. We first obtained both discrete and continuous annotations of our data. Discrete descriptors include non-verbal cues such as type of gestures, arms rest poses, head movements and smiles. Continuous descriptors concern annotators' judgments of the expert's warmth and competence during the observed interaction with the novice. Then we computed Odds Ratios between those descriptors. Results highlight the role of smiling in warmth and competence impressions. Smiling is associated with increased levels of warmth and decreasing competence. It also affects the impact of others non-verbal cues (e.g. self-adaptors gestures) on warmth and competence. Moreover, our findings provide interesting insights about the role of rest poses, that are associated with decreased levels of warmth and competence impressions.

We present a novel multi-lingual database of natural dyadic novice-expert interactions, named NoXi, featuring screen-mediated dyadic human interactions in the context of information exchange and retrieval. NoXi is designed to provide spontaneous interactions with emphasis on adaptive behaviors and unexpected situations (e.g. conversational interruptions). A rich set of audio-visual data, as well as continuous and discrete annotations are publicly available through a web interface. Descriptors include low level social signals (e.g. gestures, smiles), functional descriptors (e.g. turn-taking, dialogue acts) and interaction descriptors (e.g. engagement, interest, and fluidity).

Abstract Inverse kinematics (IK) equations are usually solved through approximated linearizations or heuristics. These methods lead to character animations that are unnatural looking or unstable because they do not consider both the motion coherence and limits of human joints. In this paper, we present a method based on the formulation of multi‐variate Gaussian distribution models (MGDMs), which precisely specify the soft joint constraints of a kinematic skeleton. Each distribution model is described by a covariance matrix and a mean vector representing both the joint limits and the coherence of motion of different limbs. The MGDMs are automatically learned from the motion capture data in a fast and unsupervised process. When the character is animated or posed, a Gaussian process synthesizes a new MGDM for each different vector of target positions, and the corresponding objective function is solved with Jacobian‐based IK. This makes our method practical to use and easy to insert into pre‐existing animation pipelines. Compared with previous works, our method is more stable and more precise, while also satisfying the anatomical constraints of human limbs. Our method leads to natural and realistic results without sacrificing real‐time performance.

It has been well documented that laughter is an important communicative and expressive signal in face-to-face conversations. Our work aims at building a laughter behavior controller for a virtual character which is able to generate upper body animations from laughter audio given as input. This controller relies on the tight correlations between laughter audio and body behaviors. A unified continuous-state statistical framework, inspired by Kalman filter, is proposed to learn the correlations between laughter audio and head/torso behavior from a recorded laughter human dataset. Due to the lack of shoulder behavior data in the recorded human dataset, a rule-based method is defined to model the correlation between laughter audio and shoulder behavior. In the synthesis step, these characterized correlations are rendered in the animation of a virtual character. To validate our controller, a subjective evaluation is conducted where participants viewed the videos of a laughing virtual character. It compares the animations of a virtual character using our controller and a state of the art method. The evaluation results show that the laughter animations computed with our controller are perceived as more natural, expressing amusement more freely and appearing more authentic than with the state of the art method.

In order to improve the social capabilities of embodied conversational agents, we propose a computational model to enable agents to automatically select and display appropriate smiling behavior during human--machine interaction. A smile may convey different communicative intentions depending on subtle characteristics of the facial expression and contextual cues. To construct such a model, as a first step, we explore the morphological and dynamic characteristics of different types of smiles (polite, amused, and embarrassed smiles) that an embodied conversational agent may display. The resulting lexicon of smiles is based on a corpus of virtual agents’ smiles directly created by users and analyzed through a machine-learning technique. Moreover, during an interaction, a smiling expression impacts on the observer’s perception of the interpersonal stance of the speaker. As a second step, we propose a probabilistic model to automatically compute the user’s potential perception of the embodied conversational agent’s social stance depending on its smiling behavior and on its physical appearance. This model, based on a corpus of users’ perceptions of smiling and nonsmiling virtual agents, enables a virtual agent to determine the appropriate smiling behavior to adopt given the interpersonal stance it wants to express. An experiment using real human--virtual agent interaction provided some validation of the proposed model.

Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

The goal of this work is to model a virtual character able to converse with different interpersonal attitudes. To build our model, we rely on the analysis of multimodal corpora of non-verbal behaviors. The interpretation of these behaviors depends on how they are sequenced (order) and distributed over time. To encompass the dynamics of non-verbal signals across both modalities and time, we make use of temporal sequence mining. Specifically, we propose a new algorithm for temporal sequence extraction. We apply our algorithm to extract temporal patterns of non-verbal behaviors expressing interpersonal attitudes from a corpus of job interviews. We demonstrate the efficiency of our algorithm in terms of significant accuracy improvement over the state-of-the-art algorithms.

We present a demonstration of the ARIA framework, a modular approach for rapid development of virtual humans for information retrieval that have linguistic, emotional, and social skills and a strong personality. We demonstrate the framework's capabilities in a scenario where `Alice in Wonderland', a popular English literature book, is embodied by a virtual human representing Alice. The user can engage in an information exchange dialogue, where Alice acts as the expert on the book, and the user as an interested novice. Besides speech recognition, sophisticated audio-visual behaviour analysis is used to inform the core agent dialogue module about the user's state and intentions, so that it can go beyond simple chat-bot dialogue. The behaviour generation module features a unique new capability of being able to deal gracefully with interruptions of the agent.

Virtual Reality and immersive experiences, which allow players to share the same virtual environment as the characters of a virtual world, have gained more and more interest recently. In order to conceive these immersive virtual worlds, one of the challenges is to give to the characters that populate them the ability to express behaviors that can support the immersion. In this work, we propose a model capable of controlling and simulating a conversational group of social agents in an immersive environment. We describe this model which has been previously validated using a regular screen setting and we present a study for measuring whether users recognized the attitudes expressed by virtual agents through the realtime generated animations of nonverbal behavior in an immersive setting. Results mirrored those of the regular screen setting thus providing further insights for improving players experiences by integrating them into immersive simulated group conversations with characters that express different interpersonal attitudes.

In the everyday world people form small conversing groups where social interaction takes place, and much of the social behavior takes place through managing interpersonal space (i.e., proxemics) and group formation, signaling their attentio to others (i.e., through gaze behavior), and expressing certain attitudes, for example, friendliness, by smiling, getting close through increased engagement and intimacy, and welcoming newcomers. Many real-time interactive systems feature virtual anthropomorphic characters in order to simulate conversing groups and add plausibility and believability to the simulated environments. However, only a few have dealt with autonomous behavior generation, and in those cases, the agents’ exhibited behavior should be evaluated by users in terms of appropriateness, believability, and conveyed meaning (e.g., attitudes). In this article we present an integrated intelligent interactive system for generating believable nonverbal behavior exhibited by virtual agents in small simulated group conversations. The produced behavior supports group formation management and the expression of interpersonal attitudes (friendly vs. unfriendly) both among the agents in the group (i.e., in-group attitude) and towards an approaching user in an avatar-based interaction (out-group attitude). A user study investigating the effects of these attitudes on users’ social presence evaluation and proxemics behavior (with their avatar) in a three-dimensional virtual city environment is presented. We divided the study into two trials according to the task assigned to users, that is, joining a conversing group and reaching a target destination behind the group. Results showed that the out-group attitude had a major impact on social presence evaluations in both trials, whereby friendly groups were perceived as more socially rich. The user’s proxemics behavior depended on both out-group and in-group attitudes expressed by the agents. Implications of these results for the design and implementation of similar intelligent interactive systems for the autonomous generation of agents’ multimodal behavior are briefly discussed.

In our demo, LoL, a user interacts with a virtual agentable to copy and to adapt its laughing and expressive behaviorson-the-fly. Our aim is to study copying capabilitiesparticipate in enhancing user’s experience in the interaction.User listens to funny audio stimuli in the presenceof a laughing agent: when funniness of audio increases, theagent laughs and the quality of its body movement (directionand amplitude of laughter movements) is modulated on-theflyby user’s body features.

In this paper, we propose a computational model that provides an Embodied Conversational Agent (ECA) with the ability to generate verbal other-repetition (repetitions of some of the words uttered in the previous user speaker turn) when interacting with a user in a museum setting. We focus on the generation of other-repetitions expressing emotional stances in appreciation sentences. Emotional stances and their semantic features are selected according to the user's verbal input, and ECA's utterance is generated according to these features. We present an evaluation of this model through users' subjective reports. Results indicate that the expression of emotional stances by the ECA has a positive effect oIn this paper, we propose a computational model that provides an Embodied Conversational Agent (ECA) with the ability to generate verbal other-repetition (repetitions of some of the words uttered in the previous user speaker turn) when interacting with a user in a museum setting. We focus on the generation of other-repetitions expressing emotional stances in appreciation sentences. Emotional stances and their semantic features are selected according to the user's verbal input, and ECA's utterance is generated according to these features. We present an evaluation of this model through users' subjective reports. Results indicate that the expression of emotional stances by the ECA has a positive effect on user engagement, and that ECA's behaviours are rated as more believable by users when the ECA utters other-repetitions.n user engagement, and that ECA's behaviours are rated as more believable by users when the ECA utters other-repetitions.

In the context of emotional body expression, previous works mainly focused on perceptual studies to identify the most important expressive cues. Only few studies gave insights on which body cues could be relevant for the classification and the characterization of emotions expressed in body movement. In this paper, we present our Random Forest based feature selection approach for the identification of relevant expressive body cues in the context of emotional body expression classification. We also discuss the ranking of relevant body cues according to each expressed emotion across a set of daily actions.

We give an overview of engagement in human-agent interaction. We discuss the different definitions of engagement in human and social science, specify how they relate to certain other concepts, and give an overview of the high level behaviour that is often associated with engagement. This work serves to position our future research on engagement in human-agent interaction.

In this paper, we study perception of intensity in-congruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and non-verbal behaviours required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and non-verbal behaviour, since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on non-verbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling etc. We first summarise three prototype versions of the SAL scenario, in which the behaviour of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analysing and synthesising the respective behaviours. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behaviour, dialogue management, and synthesis of speaker and listener behaviour of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse, and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

User interaction with Embodied Conversational Agents (ECA) should involve a significant affective component to achieve realism in communication. This aspect has been studied through different frameworks describing the relationship between user and ECA, for instance alignment, rapport and empathy. We conducted an experiment to explore how an ECA's non-verbal expression can be controlled to respond to a single affective dimension generated by users as input. Our system is based on the mapping of a high-level affective dimension, approach/avoidance, onto a new ECA control mechanism in which Action Units (AU) are activated through a neural network. Since 'approach' has been associated to prefrontal cortex activation, we use a measure of prefrontal cortex left-asymmetry through fNIRS as a single input signal representing the user's attitude towards the ECA. We carried out the experiment with 10 subjects, who have been instructed to express a positive mental attitude towards the ECA. In return, the ECA facial expression would reflect the perceived attitude under a neurofeedback paradigm. Our results suggest that users are able to successfully interact with the ECA and perceive its response as consistent and realistic, both in terms of ECA responsiveness and in terms of relevance of facial expressions. From a system perspective, the empirical calibration of the network supports a progressive recruitment of various AUs, which provides a principled description of the ECA response and its intensity. Our findings suggest that complex ECA facial expressions can be successfully aligned with one high-level affective dimension. Furthermore, this use of a single dimension as input could support experiments in the fine-tuning of AU activation or their personalization to user preferred modalities.

This article focuses on the identification and perception of facial action units displayed alone as well as the meaning decoding and perception of full-face synthesized expressions of laughter. We argue that the adequate representation of single action units is important in the decoding and perception of full-face expressions. In particular, we focus on three factors that may influence the identification and perception of single actions and full-face expressions: their presentation mode (static vs. dynamic), their intensity, and the presence of wrinkles. For the purpose of this study, we used a hybrid approach for animation synthesis that combines data-driven and procedural animations with synthesized wrinkles generated using a bump mapping method. Using such animation technique, we created animations of single action units and full-face movements of two virtual characters. Next, we conducted two studies to evaluate the role of presentation mode, intensity, and wrinkles in single actions and full-face context-free expressions. Our evaluation results show that intensity and presentation mode influence (1) the identification of single action units and (2) the perceived quality of the animation. At the same time, wrinkles (3) are useful in the identification of a single action unit and (4) influence the perceived meaning attached to the animation of full-face expressions. Thus, all factors are important for successful communication of expressions displayed by virtual characters.

Virtual worlds are more and more populated with autonomous conversational agents embodying different roles like tutor, guide, or personal assistant. In order to create more engaging and natural interactions, these agents should be endowed with social capabilities such as expressing different social attitudes through their behaviors. In this paper, we present the architecture of a socio-conversational agent composed of communicative components to detect and respond verbally and non-verbally to the user's speech and to convey different social attitudes. This paper presents the main components of this architecture. These descrpitions are illustrated with scenarios of interaction.

In this paper we focus on three aspects of multimodal expressions of laughter. First, we propose a procedural method to synthesize rhythmic body movements of laughter based on spectral analysis of laughter episodes. For this purpose, we analyze laughter body motions from motion capture data and we reconstruct them with appropriate harmonics. Then we reduce the parameter space to two dimensions. These are the inputs of the actual model to generate a continuum of laughs rhythmic body movements.

This paper presents the analysis of the multimodal behavior of experienced practitioners of job interview coaching, and describes a methodology to specify their behavior in Embodied Conversational Agents acting as virtual recruiters displaying different interpersonal stances. In a first stage, we collect a corpus of videos of job interview enactments, and we detail the coding scheme used to encode multimodal behaviors and contextual information. From the annotations of the practitioners' behaviors we observe specificities of behavior across different levels, namely monomodal behavior variations, inter-modalities behavior influences, and contextual influences on behavior. Finally we propose the adaptation of an existing agent architecture to model these specificities in a virtual recruiter's behavior.

In this paper, we explore the emotions and interpersonal stances that the expressions of smile may convey by analyzing the user's perception of smiling embodied conversational agents at different levels: (1) a signal level considering the emotions and stances that a signal of smile may convey depending on its morphological and dynamic characteristics, (2) a communicative level by exploring the effects of the ECA's smiling behavior on the stances perceived by a user, and (3) an interactive level by showing the influence of the alignment of smiles in a dyad of virtual characters on the perceived stances. In the light of this multi-level analysis of smiles, we propose the architecture of a fully interactive smiling ECA based on an extension of the SAIBA framework.

Nonverbal communicative behaviors during speech are important to model a virtual agent able to sustain a natural and lively conversation with humans. We investigate statistical frameworks for learning the correlation between speech prosody and eyebrow motion features. Such methods may be used to synthesize automatically accurate eyebrow movements from synchronized speech.

This paper presents a multimodal fuzzy inference system for emotion detection. The system extracts and merges visual, acoustic and context relevant features. The experiments have been performed as part of the AVEC 2012 challenge. Facial expressions play an important role in emotion detection. However, having an automatic system to detect facial emotional expressions on unknown subjects is still a challenging problem. Here, we propose a method that adapts to the morphology of the subject and that is based on an invariant representation of facial expressions. Our method relies on 8 key expressions of emotions of the subject. In our system, each image of a video sequence is defined by its relative position to these 8 expressions. These 8 expressions are synthesized for each subject from plausible distortions learnt on other subjects and transferred on the neutral face of the subject. Expression recognition in a video sequence is performed in this space with a basic intensity-area detector. The emotion is described in the 4 dimensions: valence, arousal, power and expectancy. The results show that the duration of high intensity smile is an expression that is meaningful for continuous valence detection and can also be used to improve arousal detection. The main variations in power and expectancy are given by context data.

Convincing conversational agents require a coherent set of behavioral responses that can be interpreted by a human observer as indicative of a personality. This paper discusses the continued development and subsequent evaluation of virtual agents based on sound psychological principles. We use Eysenck's theoretical basis to explain aspects of the characterization of our agents, and we describe an architecture where personality affects the agent's global behavior quality as well as their back-channel productions. Drawing on psychological research, we evaluate perception of our agents' personalities and credibility by human viewers (N = 187). Our results suggest that we succeeded in validating theoretically grounded indicators of personality in our virtual agents, and that it is feasible to place our characters on Eysenck's scales. A key finding is that the presence of behavioral characteristics reinforces the prescribed personality profiles that are already emerging from the still images. Our long-term goal is to enhance agents' ability to sustain realistic interaction with human users, and we discuss how this preliminary work may be further developed to include more systematic variation of Eysenck's personality scales.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

This demonstration aims to showcase the recently completed SEMAINE system. The SEMAINE system is a publicly available, fully autonomous Sensitive Artificial Listeners (SAL) system that consists of virtual dialog partners based on audiovisual analysis and synthesis (see http://semaine.opendfki.de/wiki). The system runs in real-time, and combines incremental analysis of user behavior, dialog management, and synthesis of speaker and listener behavior of a SAL character, displayed as a virtual agent. The SAL characters intend to engage the user in a conversation by paying attention to the user's emotions and nonverbal expressions. The characters have their own emotionally defined personality. During an interaction, the characters attempt to create an emotional workout for the user by drawing her/him towards their dominant emotion, through a combination of verbal and nonverbal expressions.

Emotional expressions play a very important role in the interaction between virtual agents and human users. In this paper, we present a new constraint-based approach to the generation of multimodal emotional displays. The displays generated with our method are not limited to the face, but are composed of different signals partially ordered in time and belonging to different modalities. We also describe the evaluation of the main features of our approach. We examine the role of multimodality, sequentiality, and constraints in the perception of synthesized emotional states. The results of our evaluation show that applying our algorithm improves the communication of a large spectrum of emotional states, while the believability of the agent animations increases with the use of constraints over the multimodal signals.

This work presents the development of a real-time framework for the research of multimodal feedback of robots/talking agents in the context of Human Robot Interaction (HRI) and Human Computer Interaction (HCI). For evaluating the framework, a Multimodal corpus is built (ENTERFACE_STEAD), and a study on the important multimodal features was done for building an active Robot/Agent listener of a storytelling experience with Humans. The experiments show that even when building the same reactive behavior models for Robot and Talking Agents, the interpretation and the realization of the behavior communicated is different due to the different communicative channels Robots/Agents offer be it physical but less human-like in Robots, and virtual but more expressive and human-like in Talking agents.

In this paper, we present a system that visualizes the expressive quality of a music performance using a virtual head. We provide a mapping through several parameter spaces: on the input side, we have elaborated a mapping between values of acoustic cues and emotion as well as expressivity parameters; on the output side, we propose a mapping between these parameters and the behaviors of the virtual head. This mapping ensures a coherency between the acoustic source and the animation of the virtual head. After presenting some background information on behavior expressivity of humans, we introduce our model of expressivity. We explain how we have elaborated the mapping between the acoustic and the behavior cues. Then, we describe the implementation of a working system that controls the behavior of a human-like head that varies depending on the emotional and acoustic characteristics of the musical execution. Finally, we present the tests we conducted to validate our mapping between the emotive content of the music performance and the expressivity parameters.

This project aims at the creation of a virtual humanoid endowed with expressive gestures. More specifically, we focus our attention on expressiveness (what type of gesture: fluidity, tension, anger) and on its semantic representations. Our approach relies on a data-driven animation scheme. From motion data captured thanks to an optical system and data gloves, we try to extract significant features of communicative gestures, and to re-synthesize them afterward with style variation. The proposed model is applied to the generation of a set of French sign language (FSL) gestures. Within this framework, a database involving the whole body, hands motion and facial expressions has been built The analysis of this database makes possible information retrieval about the semantics as well as the execution style of FSL gestures. These characteristics are integrated in gesture synthesis models qualitatively evaluated by their intelligibility and the realism of the produced animations.

Man-machine interaction (MMI) systems that utilize multimodal information about users' current emotional state are presently at the forefront of interest of the computer vision and artificial intelligence communities. A lifelike avatar can enhance interactive applications. In this paper, we present the implementation of GretaEngine and synthesized expressions, including intermediate ones, based on MPEG-4 standard and Whissel's emotion representation.

In this paper we present our work toward the creation of a multimodal expressive Embodied Conversational Agent (ECA). Our agent, called Greta, exhibits nonverbal behaviors synchronized with speech. We are using the taxonomy of communicative functions developed by Isabella Poggi [22] to specify the behavior of the agent. Based on this taxonomy a representation language, Affective Presentation Markup Language, APML has been defined to drive the animation of the agent [4]. Lately, we have been working on creating no longer a generic agent but an agent with individual characteristics. We have been concentrated on the behavior specification for an individual agent. In particular we have defined a set of parameters to change the expressivity of the agent's behaviors. Six parameters have been defined and implemented to encode gesture and face expressivity. We have performed perceptual studies of our expressivity model.

Abstract We aim at the realization of an Embodied Conversational Agent able to interact naturally and emotionally with user. In particular, the agent should behave expressively. Specifying for a given emotion, its corresponding facial expression will not produce the sensation of expressivity. To do so, one needs to specify parameters such as intensity, tension, movement property. Moreover, emotion affects also lip shapes during speech. Simply adding the facial expression of emotion to the lip shape does not produce lip readable movement. In this paper we present a model based on real data from a speaker on which was applied passive markers. The real data covers natural speech as well as emotional speech. We present an algorithm that determines the appropriate viseme and applies coarticulation and correlation rules to consider the vocalic and the consonantal contexts as well as muscular phenomena such as lip compression and lip stretching. Expressive qualifiers are then used to modulate the expressivity of lip movement. Our model of lip movement is applied on a 3D facial model compliant with MPEG‐4 standard. Copyright © 2004 John Wiley & Sons, Ltd.

In this paper we present some results to model complex interactions among virtual characters that participate in negotiation dialogues as well as our work related to a gaze model that controls the eye behavior of several agents conversing with each other. As a test-bed we have created an Avatar Arena in which several avatars negotiate on meeting arrangement tasks on behalf of their users. To enhance the naturalness of the emerging negotiation dialogues we need to determine both the behaviors of speakers as well as the behaviors of listening characters. In our approach we try to exploit socio-physiological concepts, such as cognitive balance and dissonance to determine verbal and nonverbal behavior of all dialogue participants. We pay particular attention to the gaze behavior of the speaker considering the communicative functions the speaker desires to communicate as well as a statistical model of eye movements. In addition, a model of gaze behavior for listeners is also proposed.

This paper introduces Gesture Engine, an animation system that synthesizes human gesturing behaviors from augmented conversation transcripts using a database of highlevel gesture definitions. An abstract scripting language to specify hand-arm gestures is introduced that incorporates knowledge from sign language research, psycholinguistics, and traditional keyframe animation. A new planning algorithm instantiates and adjusts gestures according to communicative context and temporal constraints obtained from a speech synthesizer The system animates an MPEG-4 compliant skeleton using Body Animation Parameters.

Abstract Our goal is to develop a believable embodied agent able to dialogue with a user. In particular, we aim at making an agent that can also combine facial expressions in a complex and subtle way, just like a human agent does. We first review a taxonomy of communicative functions that our agent is able to express non‐verbally; but we point out that, due to the complexity of communication, in some cases different information can be provided at once by different parts and actions of an agent's face. In this paper we are interested in assessing and treating what happens, at the meaning and signal levels of behaviour, when different communicative functions have to be displayed at the same time and necessarily have to make use of the same expressive resources. In some of these cases the complexity of the agent's communication can give rise to conflicts between the parts or movements of the face. In this paper, we propose a way to manage the possible conflicts between different modalities of communication through the tool of belief networks, and we show how this tool allows us to combine facial expressions of different communicative functions and to display complex and subtle expressions. Copyright © 2002 John Wiley & Sons, Ltd.

Our goal is to create a natural talking face with, in particular, lip-readable movements. Based on real data extracted from an Italian speaker with the ELITE system, we have approximated the data using radial basis functions. In this paper we present our 3D facial model based on MPEG-4 standard and our computational model of lip movements for Italian. Our experiment is based on some phonetic-phonological considerations on the parameters defining labial orifice, and on identification tests of visual articulatory movements.

Dialoging with a synthetic agent is a vast research topic to enhance user-interface friendliness. We present in this paper an on-going project on the simulation of a dialog situation between two synthetic agents. More particularly we focus our interest on finding the appropriate facial expressions of a speaker addressing to different types of listeners (tourist, employee, child, and so on) using various linguistic forms such as request, question, information. Communication between speaker and listener involves multimodal behaviors such as the choice of words, intonation and paralinguistic parameters for the vocal ones; facial expressions, gaze, gesture and body movements for the non-verbal ones. The choice of each individual behavior, their mutual interaction and synchronization produce the richness and subtility of human communication.In order to develop a system that computes automatically the appropriate facial and gaze behaviors corresponding to a communicative act for a given speaker and listener, our first step is to categorize facial expressions and gaze based on their communicative functions rather than on their appearance. The next step is to find inference rules that describe the mental process ongoing in the speaker while communicating with the listener. The rules take into account the power relation between speaker and listener and the beliefs the speaker has about the listener to constrain the choice of performative acts.

A geometric and kinematic model for describing the global shape and the predominant motions of the human tongue, to be applied in computer animation, is discussed. The model consists of a spatial configuration of moving points that form the vertices of a mesh of 9 3-D triangles. These triangles are interpreted as charge centres (the so-called skeleton) for a potential field, and the surface of the tongue is modelled as an equi-potential surface of this field. In turn, this surface is approximated by a triangular mesh prior to rendering. As to the motion of the skeleton, precautions are taken in order to achieve (approximate) volume conservation; the computation of the triangular mesh describing the surface of the tongue implements penetration avoidance with respect to the palate. Further, the motions of the skeleton derive from a formal speech model which also controls the motion of the lips to arrive at a visually plausible speech synchronous mouth model.>

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversation is created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gestures generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout we will use examples from an actual synthesized, fully animated conversation.