Conversational grounding is the collaborative process through which speakers establish and maintain mutual understanding. It is essential for the success of a dialogue. While it is inherent in human conversations, it remains a challenge for instruction-following Large Language Models (LLM). This paper surveys how conversational grounding is evaluated in task-oriented dialogue in the current era of LLMs. First, we focus on how conversational grounding is modelled explicitly — using dialogue acts and by modelling the participant mental state. Then, we review collaborative tasks that enable the evaluation of conversational grounding implicitly at the global level based on outcomes. Finally, we highlight the limitations of evaluation, notably the current methodology and metrics used, and outline research directions in conversational grounding and its evaluation.

Evaluating the quality of generated nonverbal behavior remains a major challenge in the development of generative models.While human evaluations are reliable, they are costly and impractical for large-scale or iterative optimization.In this work, we propose an objective evaluation framework based on aggregated ranks across multiple fidelity and diversity metrics, computed from both raw features and learned latent representations.Compared to existing works, our framework emphasizes consistency across multiple metrics, aiming to provide a more holistic assessment.

Public speaking (PS) self-assessment remains challenging due to multiple behavioral dimensions and a lack of objective evaluation methods.Virtual reality (VR) offers immersive training environments with automatic performance analysis capabilities.However, current evaluation systems use ad hoc metrics lacking transparency and reproducibility.No comprehensive set of multimodal, contextindependent behavioral cues exists for interpretable user feedback.We propose verbal and nonverbal features meeting three criteria: automatic measurement capability, context independence, and user interpretability.Using a multimodal corpus of VR presentations by 60 participants, we extracted 47 behavioral features via the Meta Quest Pro headset.Expert assessment used 7-point Likert scales.Correlation analysis and machine learning models demonstrate this feature set provides a relevant basis for automated PS assessment in VR.

In a virtual reality public speaking training system, it is essential to control the audience’s nonverbal behavior in order to simulate different attitudes. The virtual audience’s social attitude is generally represented by a two-dimensional valence-arousal model describing the opinion and engagement of virtual characters. In this article, we argue that the valence-arousal representation is not sufficient to describe the user’s perception of a virtual character’s social attitude. We propose a three-dimensional model by dividing the valence axis into two dimensions representing the epistemic and affective stance of the virtual character, reflecting the character’s agreement and emotional reaction. To assess the perception of the virtual characters’ nonverbal behavior on these two new dimensions, we conducted a perceptive study in virtual reality with 44 participants who evaluated 50 animations combining multimodal nonverbal behavioral signals such as head movements, facial expressions, gaze direction and body posture. The results of our experiment show that, in fact, the valence axis should be divided into two axes to take into account the perception of the virtual character’s epistemic and affective stance. Furthermore, the results show that one behavioral signal is predominant for the evaluation of each dimension: head movements for the epistemic dimension and facial expressions for the affective dimension. These results provide useful guidelines for designing the nonverbal behavior of a virtual audience for social attitudes’ simulation.

In the realm of human communication, feedback plays a pivotal role in shaping the dynamics of conversations. This study delves into the multifaceted relationship between listener feedback, narration quality and distraction effects. We present an analysis conducted on the SMYLE corpus, specifically enriched for this study, where 30 dyads of participants engaged in 1) face-to-face storytelling (8.2 hours) followed by 2) a free conversation (7.8 hours). The storytelling task unfolds in two conditions, where a storyteller engages with either a “normal” or a “distracted” listener. Examining the feedback impact on storytellers, we discover a positive correlation between the frequency of specific feedback and the narration quality in normal conditions, providing an encouraging conclusion regarding the enhancement of interaction through specific feedback in distraction-free settings. In contrast, in distracted settings, a negative correlation emerges, suggesting that increased specific feedback may disrupt narration quality, underscoring the complexity of feedback dynamics in human communication. The contribution of this paper is twofold: first presenting a new and highly enriched resource for the analysis of discourse phenomena in controlled and normal conditions; second providing new results on feedback production, its form and its consequence on the discourse quality (with direct applications in human-machine interaction).

International audience

Research on non-verbal behavior generation for social interactive agents focuses mainly on the believability and synchronization of non-verbal cues with speech. However, existing models, predominantly based on deep learning architectures, often perpetuate biases inherent in the training data. This raises ethical concerns, depending on the intended application of these agents. This paper addresses these issues by first examining the influence of gender on facial non-verbal behaviors. We concentrate on gaze, head movements, and facial expressions. We introduce a classifier capable of discerning the gender of a speaker from their non-verbal cues. This classifier achieves high accuracy on both real behavior data, extracted using state-of-the-art tools, and synthetic data, generated from a model developed in previous work. Building upon this work, we present a new model, FairGenderGen, which integrates a gender discriminator and a gradient reversal layer into our previous behavior generation model. This new model generates facial non-verbal behaviors from speech features, mitigating gender sensitivity in the generated behaviors. Our experiments demonstrate that the classifier, developed in the initial phase, is no longer effective in distinguishing the gender of the speaker from the generated non-verbal behaviors.

In this article, we present a new open-source tool, called OUTCOME Virtual, to automatically replicate humans’ behaviors on virtual characters. The tool takes as input a human video and extracts action units, head movements, and gaze direction to animate a virtual character’s face and head with the synchronized speech in Unity. The tool is designed to be easily configured, in particular concerning the association between actions units and blend shapes, and with a user-friendly interface to select the appearance of virtual characters, the lip synchronization and the noise-correction option. The tool is illustrated in this paper with the generation of listener feedback.

The multimodal behaviour of IVAs may convey different socio-affective dimensions, such as emotions, personality, or social capabilities. Several research works show that factors may impact the perception of the IVA’s behaviour. This paper proposes a systematic review, based on the PRISMA method, to investigate how the multimodal behaviour of IVAs is perceived with respect to socio-affective dimensions. To compare the results of different research works, a socio-emotional framework is proposed, considering the dimensions commonly employed in the studies. The conducted analysis of a wide array of studies ensures a comprehensive and transparent review, providing guidelines on the design of socio-affective IVAs.

In this article, we present a new tool called REVITALISE (viRtual bEhaVioral skIlls TrAining for pubLIc SpEaking) which enables the user to practice public speaking in front of a virtual audience in virtual reality. The tool is specially designed to simulate different virtual audiences to vary the user experiences. The virtual environments as well as the genders of the virtual characters and their non-verbal behaviors can be easily set-up in REVITALISE. Moreover, the motion capture-based animations, validated through a perceptive studies, enable to simulate and combine different social attitudes of the virtual audience.

To create conversational systems with humanlike listener behavior, generating short feedback responses (e.g., "mhm", "ah", "wow") appropriate for their context is crucial.These responses convey their communicative function through their lexical form and their prosodic realization.In this paper, we transplant the prosody of feedback responses from humanhuman U.S. English telephone conversations to a target speaker using two synthesis techniques (TTS and signal processing).Our evaluation focuses on perceived naturalness, contextual appropriateness and preservation of communicative function.Results indicate TTS-generated feedback were perceived as more natural than signal-processing-based feedback, with no significant difference in appropriateness.However, the TTS did not consistently convey the communicative function of the original feedback.

This study investigates conversational feedback, that is, a listener's reaction in response to a speaker, a phenomenon which occurs in all natural interactions. Feedback depends on the main speaker's productions and in return supports the elaboration of the interaction. As a consequence, feedback production has a direct impact on the quality of the interaction. This paper examines all types of feedback, from generic to specific feedback, the latter of which has received less attention in the literature. We also present a fine-grained labeling system introducing two sub-types of specific feedback: positive/negative and given/new. Following a literature review on linguistic and machine learning perspectives highlighting the main issues in feedback prediction, we present a model based on a set of multimodal features which predicts the possible position of feedback and its type. This computational model makes it possible to precisely identify the different features in the speaker's production (morpho-syntactic, prosodic and mimo-gestural) which play a role in triggering feedback from the listener; the model also evaluates their relative importance. The main contribution of this study is twofold: we sought to improve 1/ the model's performance in comparison with other approaches relying on a small set of features, and 2/ the model's interpretability, in particular by investigating feature importance. By integrating all the different modalities as well as high-level features, our model is uniquely positioned to be applied to French corpora.

There has been a lot of work on predicting the timing of feedback in conversational systems. However, there has been less focus on predicting the prosody and lexical form of feedback given their communicative function. Therefore, in this paper we present our preliminary annotations of the communicative functions of 1627 short feedback tokens from the Switchboard corpus and an analysis of their lexical realizations and prosodic characteristics. Since there is no standard scheme for annotating the communicative function of feedback we propose our own annotation scheme. Although our work is ongoing, our preliminary analysis revealed lexical tokens such as “yeah” are ambiguous and therefore lexical forms alone are not indicative of the function. Both the lexical form and prosodic characteristics need to be taken into account in order to predict the communicative function. We also found that feedback functions have distinguishable prosodic characteristics in terms of duration, mean pitch, pitch slope, and pitch range.

In the well-known Turing test, humans have to judge whether they write to another human or a chatbot. In this article, we propose a reversed Turing test adapted to live conversations: based on the speech of the human, we have developed a model that automatically detects whether she/he speaks to an artificial agent or a human. We propose in this work a prediction methodology combining a step of specific features extraction from behaviour and a specific deep learning model based on recurrent neural networks. The prediction results show that our approach, and more particularly the considered features, improves significantly the predictions compared to the traditional approach in the field of automatic speech recognition systems, which is based on spectral features, such as Mel-frequency Cepstral Coefficients (MFCCs). Our approach allows evaluating automatically the type of conversational agent, human or artificial agent, solely based on the speech of the human interlocutor. Most importantly, this model provides a novel and very promising approach to weigh the importance of the behaviour cues used to make correctly recognize the nature of the interlocutor, in other words, what aspects of the human behaviour adapts to the nature of its interlocutor.

Dialogue Act classification becomes a complex task when dealing with fine-grain labels.Many applications require such level of labelling, typically automatic dialogue systems.We present in this paper a 2-level classification technique, distinguishing between generic and specific dialogue acts (DA).This approach makes it possible to benefit from the very good accuracy of generic DA classification at the first level and proposes an efficient approach for specific DA, based on high-level linguistic features.Our results show the interest of involving such features into the classifiers, outperforming all other feature sets, in particular those classically used in DA classification.

International audience

International audience

In this paper, we present a tool allowing dynamic prediction and visualization of an individual’s local brain activity during a conversation. The prediction module of this tool is based on classifiers trained using a corpus of human-human and human-robot conversations including fMRI recordings. More precisely, the module takes as input behavioral features computed from raw data, mainly the participant and the interlocutor speech but also the participant’s visual input and eye movements. The visualisation module shows in real-time the dynamics of brain active areas synchronised with the behavioral raw data. In addition, it shows which integrated behavioral features are used to predict the activity in individual brain areas.

This paper presents an original dataset of controlled interactions, focusing on the study of feedback items. It consists on recordings of different conversations between a doctor and a patient, played by actors. In this corpus, the patient is mainly a listener and produces different feedbacks, some of them being (voluntary) incongruent. Moreover, these conversations have been re-synthesized in a virtual reality context, in which the patient is played by an artificial agent. The final corpus is made of different movies of human-human conversations plus the same conversations replayed in a human-machine context, resulting in the first human-human/human-machine parallel corpus. The corpus is then enriched with different multimodal annotations at the verbal and non-verbal levels. Moreover, and this is the first dataset of this type, we have designed an experiment during which different participants had to watch the movies and give an evaluation of the interaction. During this task, we recorded participant’s brain signal. The Brain-IHM dataset is then conceived with a triple purpose: 1/ studying feedbacks by comparing congruent vs. incongruent feedbacks 2/ comparing human-human and human-machine production of feedbacks 3/ studying the brain basis of feedback perception.

In this paper we present investigation of real-life, bi-directional conversations. We introduce the multimodal corpus derived from these natural conversations alternating between human-human and human-robot interactions. The human-robot interactions were used as a control condition for the social nature of the human-human conversations. The experimental set up consisted of conversations between the participant in a functional magnetic resonance imaging (fMRI) scanner and a human confederate or conversational robot outside the scanner room, connected via bidirectional audio and unidirectional videoconferencing (from the outside to inside the scanner). A cover story provided a framework for natural, real-life conversations about images of an advertisement campaign. During the conversations we collected a multimodal corpus for a comprehensive characterization of bi-directional conversations. In this paper we introduce this multimodal corpus which includes neural data from functional magnetic resonance imaging (fMRI), physiological data (blood flow pulse and respiration), transcribed conversational data, as well as face and eye-tracking recordings. Thus, we present a unique corpus to study human conversations including neural, physiological and behavioral data.

A key challenge when studying human-agent interaction is the evaluation of user's experience. In virtual reality, this question is addressed by studying the sense of "presence'' and"co-presence'', generally assessed thanks to well-grounded subjective post-experience questionnaires. In this article, we aim at exploring behavioral measures of presence and co-presence by analyzing multimodal cues produced during an interaction both by the user and the virtual agent. In our study, we started from a corpus of human-agent interaction collected in a task-oriented context: a virtual environment aiming at training doctors to break bad news to a patient (played by a virtual agent). Based on this corpus, we have used machine learning algorithms to explore the possibility of predicting user's sense of presence and co-presence. In particular, we have applied and compared two techniques, Random forest and SVM, both showing very good results in predicting the level of presence and co-presence.

In the intelligent virtual agent domain, several machine learning models have been proposed to automatically determine the feedbacks of virtual agents during an interaction, using human-human interaction datasets as training corpora and most commonly based on verbal and prosodic features \citeMorency2010, Truong2010a. These approaches suppose an accurate system to automatically recognize speech and prosody. That makes the overall model's performance dependent on the individual performances of speech and prosody recognizers. As a consequence, one challenge remains to identify features that could be easily and accurately recognized during a human-machine interaction for predicting virtual agents' feedbacks in real time.

In many applications, Embodied Conversational Agents (ECAs) must be able to express various affects such as emotions or social attitudes. Non-verbal signals, such as smiles or gestures, contribute to the expression of attitudes. Social attitudes affect the whole behavior of a person: they are “characteristic of an affective style that colors the entire interaction” [1] . Moreover, recent findings have demonstrated that non-verbal signals are not interpreted in isolation but along with surrounding signals. Non-verbal behavior planning models designed to allow ECAs to express attitudes should thus consider complete sequences of non-verbal signals and not only signals independently of one another. However, existing models do not take this into account, or in a limited manner. The contribution of this paper is a methodology for the automatic extraction of sequences of non-verbal signals characteristic of a social phenomenon from a multimodal corpus, and a non-verbal behavior planning model that takes into account sequences of non-verbal signals rather than signals independently. This methodology is applied to design a virtual recruiter capable of expressing social attitudes, which is then evaluated in and out of an interaction context.

We present an approach to objectify the social competence of artificial agents using human brain neurophysiology. Whole brain activity is recorded with functional Magnetic Resonance Imaging (fMRI) while participants discuss either with a human confederate or an artificial agent. This allows a direct comparison of local brain responses, including deep brain structures invisible to other neuroimaging techniques, as a function of the nature of the interlocutor. The present data (9 participants, artificial agent is the robotic conversational head Furhat controlled with a Wizard of Oz procedure) demonstrates the feasibility of this approach, and results confirm an increased activity in the hypothalamic region when interacting with a human compared to an artificial agent.

In human-agent interaction, one key challenge is the evaluation of the user's experience. In the virtual reality domain, the sense of presence and co-presence, reflecting the psychological immersion of the user, is generally assessed through well-grounded subjective post-experience questionnaires. In this article, we aim at presenting a new way to automatically predict the sense of presence and co-presence of a user at the end of an interaction based on specific verbal and non-verbal behavioral cues automatically computed. A random forest algorithm has been applied on a human-agent interaction corpus collected in the specific context of a virtual environment developed to train doctors to break bad news to a virtual patient. The performance of the models demonstrate the capacity to automatically and accurately predict the level of presence and co-presence, but also show the relevancy of the verbal and non-verbal behavioral cues as objective measures of presence.

The workshop “Investigating Social Interactions With Artificial Agents” organized within the “International Conference on Multimodal Interactions 2017” attempts to bring together researchers from different fields sharing a similar interest in human interactions with other agents. If interdisciplinarity is necessary to address the question of the “Turing Test”, namely “can an artificial conversational artificial agent be perceived as human”, it is also a very promising new way to investigate social interactions in the first place. Biology is represented by social cognitive neuroscience, aiming to describe the physiology of human social behaviors. Linguistics, from humanities, attempts to characterize a specifically human behavior and language. Social Signal Processing is a recent approach to analyze automatically, using advanced Information Technologies, the behaviors pertaining to natural human interactions. Finally, from Artificial Intelligence, the development of artificial agents, conversational and/or embodied, onscreen or physically attempts to recreate non-human socially interactive agents for a multitude of applications.

While several research works have shown that virtual agents are able to generate natural and social behaviors from users, few of them have compared these social reactions to those expressed dur- ing a human-human mediated communication. In this paper, we propose to explore the social cues expressed by a user during a mediated communication either with an embodied conversational agent or with another human. For this purpose, we have exploited a machine learning method to identify the facial and head social cues characteristics in each interaction type and to construct a model to automatically determine if the user is interacting with a virtual agent or another human. ‘e results show that, in fact, the users do not express the same facial and head movements during a communication with a virtual agent or another user. Based on these results, we propose to use such a machine learning model to automatically measure the social capability of a virtual agent to generate a social behavior in the user comparable to a human- human interaction. ‘e resulting model can detect automatically if the user is communicating with a virtual or real interlocutor, looking only at the user’s face and head during one second.

Doctors should be trained not only to perform medical or surgical acts but also to develop competences in communication for their interaction with patients. For instance, the way doctors deliver bad news has a significant impact on the therapeutic process. In order to facilitate the doctors’ training to break bad news, we aim at developing a virtual patient ables to interact in a multimodal way with doctors announcing an undesirable event. One of the key elements to create an engaging interaction is the feedbacks’ behavior of the virtual character. In order to model the virtual patient’s feedbacks in the context of breaking bad news, we have analyzed a corpus of real doctor’s training. The verbal and nonverbal signals of both the doctors and the patients have been annotated. In order to identify the types of feedbacks and the elements that may elicit a feedback, we have explored the corpus based on sequences mining methods. Rules, that have been extracted from the corpus, enable us to determine when a virtual patient should express which feedbacks when a doctor announces a bad new

In order to improve the social capabilities of embodied conversational agents, we propose a computational model to enable agents to automatically select and display appropriate smiling behavior during human--machine interaction. A smile may convey different communicative intentions depending on subtle characteristics of the facial expression and contextual cues. To construct such a model, as a first step, we explore the morphological and dynamic characteristics of different types of smiles (polite, amused, and embarrassed smiles) that an embodied conversational agent may display. The resulting lexicon of smiles is based on a corpus of virtual agents’ smiles directly created by users and analyzed through a machine-learning technique. Moreover, during an interaction, a smiling expression impacts on the observer’s perception of the interpersonal stance of the speaker. As a second step, we propose a probabilistic model to automatically compute the user’s potential perception of the embodied conversational agent’s social stance depending on its smiling behavior and on its physical appearance. This model, based on a corpus of users’ perceptions of smiling and nonsmiling virtual agents, enables a virtual agent to determine the appropriate smiling behavior to adopt given the interpersonal stance it wants to express. An experiment using real human--virtual agent interaction provided some validation of the proposed model.

Virtual Reality and immersive experiences, which allow players to share the same virtual environment as the characters of a virtual world, have gained more and more interest recently. In order to conceive these immersive virtual worlds, one of the challenges is to give to the characters that populate them the ability to express behaviors that can support the immersion. In this work, we propose a model capable of controlling and simulating a conversational group of social agents in an immersive environment. We describe this model which has been previously validated using a regular screen setting and we present a study for measuring whether users recognized the attitudes expressed by virtual agents through the realtime generated animations of nonverbal behavior in an immersive setting. Results mirrored those of the regular screen setting thus providing further insights for improving players experiences by integrating them into immersive simulated group conversations with characters that express different interpersonal attitudes.

In this paper, we present an application of symbolic data processing for the design of virtual character's smiling facial expressions. A collected database of virtual character's smiles directly created by users has been explored using symbolic data analysis methods. An unsupervised analysis has enabled us to identify the morphological and dynamic characteristics of different types of smiles as well as of combinations of smiles. Based on the symbolic data analysis, to generate different smiling faces, we have developed procedures to automatically reconstitute smiling virtual faces from a point in a multidimensional space corresponding to a principal component analysis plane.

In the everyday world people form small conversing groups where social interaction takes place, and much of the social behavior takes place through managing interpersonal space (i.e., proxemics) and group formation, signaling their attentio to others (i.e., through gaze behavior), and expressing certain attitudes, for example, friendliness, by smiling, getting close through increased engagement and intimacy, and welcoming newcomers. Many real-time interactive systems feature virtual anthropomorphic characters in order to simulate conversing groups and add plausibility and believability to the simulated environments. However, only a few have dealt with autonomous behavior generation, and in those cases, the agents’ exhibited behavior should be evaluated by users in terms of appropriateness, believability, and conveyed meaning (e.g., attitudes). In this article we present an integrated intelligent interactive system for generating believable nonverbal behavior exhibited by virtual agents in small simulated group conversations. The produced behavior supports group formation management and the expression of interpersonal attitudes (friendly vs. unfriendly) both among the agents in the group (i.e., in-group attitude) and towards an approaching user in an avatar-based interaction (out-group attitude). A user study investigating the effects of these attitudes on users’ social presence evaluation and proxemics behavior (with their avatar) in a three-dimensional virtual city environment is presented. We divided the study into two trials according to the task assigned to users, that is, joining a conversing group and reaching a target destination behind the group. Results showed that the out-group attitude had a major impact on social presence evaluations in both trials, whereby friendly groups were perceived as more socially rich. The user’s proxemics behavior depended on both out-group and in-group attitudes expressed by the agents. Implications of these results for the design and implementation of similar intelligent interactive systems for the autonomous generation of agents’ multimodal behavior are briefly discussed.

Virtual worlds are more and more populated with autonomous conversational agents embodying different roles like tutor, guide, or personal assistant. In order to create more engaging and natural interactions, these agents should be endowed with social capabilities such as expressing different social attitudes through their behaviors. In this paper, we present the architecture of a socio-conversational agent composed of communicative components to detect and respond verbally and non-verbally to the user's speech and to convey different social attitudes. This paper presents the main components of this architecture. These descrpitions are illustrated with scenarios of interaction.

This paper presents the analysis of the multimodal behavior of experienced practitioners of job interview coaching, and describes a methodology to specify their behavior in Embodied Conversational Agents acting as virtual recruiters displaying different interpersonal stances. In a first stage, we collect a corpus of videos of job interview enactments, and we detail the coding scheme used to encode multimodal behaviors and contextual information. From the annotations of the practitioners' behaviors we observe specificities of behavior across different levels, namely monomodal behavior variations, inter-modalities behavior influences, and contextual influences on behavior. Finally we propose the adaptation of an existing agent architecture to model these specificities in a virtual recruiter's behavior.

In this paper, we explore the emotions and interpersonal stances that the expressions of smile may convey by analyzing the user's perception of smiling embodied conversational agents at different levels: (1) a signal level considering the emotions and stances that a signal of smile may convey depending on its morphological and dynamic characteristics, (2) a communicative level by exploring the effects of the ECA's smiling behavior on the stances perceived by a user, and (3) an interactive level by showing the influence of the alignment of smiles in a dyad of virtual characters on the perceived stances. In the light of this multi-level analysis of smiles, we propose the architecture of a fully interactive smiling ECA based on an extension of the SAIBA framework.

One of the main challenges faced by the video game industry is to give life tobelievablenonplayer characters (NPCs). Research shows that emotions play a key role in determining the behavior of individuals. In order to improve the believability of NPCs' behavior, we propose in this paper a model of the dynamics of emotions taking into account the personality and the social relations of the character. First, we present work from the literature on emotions, personality, and social relations in computer science and in human and social sciences. We focus on the influence of personality on the triggering of emotions, and the influence of emotions on the dynamics of social relations. Based on this work, we propose a dynamic model of the socioemotional state and its implementation as part of a tool for game programmers. This tool aims at the simulation of the evolution of emotions and social relations of NPCs based on their personality and roles.