Cohesion is a multidimensional construct reflecting a shared commitment to group task(s) and strong interpersonal bonds.It plays a crucial role in collective success and emotional well-being.In human-computer interactions, embodied conversational agents can help foster group cohesion and prevent negative group dynamics by detecting and predicting low cohesion in real-time.This study introduces a novel social signal processing framework for real-time cohesion estimation in hybrid group interactions and develops computational models based on it.Using a multiparty interaction corpus, we extract nonverbal social signals available in real-time and focus on estimating overall cohesion and its dimensions (social, task).We further examine the contribution of social signals to the accuracy of low cohesion detection in collaborative interactions.In our experimental setting, overall low cohesion is more effectively detected than its social or task dimensions.The ablation study suggests that specific social signals reflecting interindividual interaction patterns substantially contribute to low cohesion detection performance.This work demonstrates the potential of data-driven methods for real-time cohesion estimation in multiparty interactions, highlighting promising opportunities for ECAs to restore and maintain cohesion in hybrid group settings.

In order to improve multi-party social interaction with artificial companions such as robots or virtual agents, real-time Emotion Recognition in Conversation (ERC) is required. In this context, ERC is a challenging task which involves multiple challenges, such as processing multimodal data over time, taking into account the multi-party context with any number of participants, understanding implied relevant commonsense knowledge during interaction and taking into account each participant’s emotional attitude. To deal with the aforementioned challenges, we design a multimodal off-the-shelf model that meets the requirements of real-life scenarios, specifically dyadic and multi-party interactions. We propose a Knowledge Aware Multi-Headed Network that integrates various sources including the dialog history and commonsense knowledge about the speaker and other participants. The weights of these pieces of information are modulated using a multi-head attention mechanism. The proposed model is learnt in a Multi-Task Learning framework which combines the ERC task with a Dialogue Act (DA) recognition task and an Emotion Shift (ES) detection task through a joint learning strategy. Our proposition obtains competitive and stable results on several benchmark datasets that vary in number of participants and length of conversations, and outperforms the state-of-the-art on one of these datasets. The importance of DA and ES prediction in determining the speaker’s current emotional state is investigated.

Empathic behavior between humans often has a positive effect, particularly in healthcare, since it facilitates relationships, improves engagement, and reduces stress and anxiety. Despite the importance of empathic communication and social relationship in healthcare, effects of empathic behavior of embodied virtual agents that interact with patients in a multimodal and adaptive way have not been widely explored. In this article, we propose an empathic model which endows a therapeutic embodied virtual agent with multi-modal adaptive empathic behavior during interaction with a user. This model relies on the user-agent interaction relationship and focuses on (1) the interpretation of user's behavior using multimodal input, and (2) the generation of multimodal empathic behavior during interaction. An experimental study in the context of empathic interaction with students during the COVID-19 pandemic is presented to evaluate the effects of adaptive empathic behavior of an agent on the quality of user interaction. Compared to an agent that relies on low-level affect matching and backchannels, results show that our agent is perceived as more empathic and improves user engagement during the interaction.

Empathic behavior between humans often has a positive effect, particularly in healthcare, since it facilitates relationship, improves engagement, and reduces stress and anxiety. Despite this importance of empathic communication and social relationship in healthcare, the effects of empathic behavior of embodied virtual agents that interact with patients in a multimodal and adaptive way have not been widely explored. The proposed multimodal adaptive empathic agent architecture (MAEAA) endows a therapeutic embodied virtual agent (EVA) with adaptive empathic behavior during interaction with a user. This architecture relies on user-agent interaction relationship and focuses on (1) the interpretation of user's behavior using multimodal input, and (2) the generation of multimodal empathic behavior during interaction. The solution have been used to develop a user study for the empathic interaction with students in the healthcare context [3].

In this article, we propose a novel routine pattern extraction architecture to analyze the daily behaviors of mobile users. The key component of the proposed architecture is a dynamic programming-based sequence dissimilarity calculation method, which aims to measure the dissimilarity between trajectories and then extract patterns using the clustering method. The method exploits three different information: (1) spatial-temporal information, (2) information in the continuous same location in the sequence and (3) the probabilities of location occurrences in the data. We conduct experiments on a synthetic dataset and two real-world datasets. The obtained results demonstrate that our proposed method is efficient in extracting hidden routine patterns from users’ trajectory data.

Arabizi is a written form of spoken Arabic, relying on Latin characters and digits.It is informal and does not follow any conventional rules, raising many NLP challenges.In particular, Arabizi has recently emerged as the Arabic language in online social networks, becoming of great interest for opinion mining and sentiment analysis.Unfortunately, only few Arabizi resources exist and state-of-the-art language models such as BERT do not consider Arabizi.In this work, we construct and release two datasets: (i) LAD, a corpus of 7.7M tweets written in Arabizi and (ii) SALAD, a subset of LAD, manually annotated for sentiment analysis.Then, a BERT architecture is pre-trained on LAD, in order to create and distribute an Arabizi language model called BAERT.We show that a language model (BAERT) pre-trained on a large corpus (LAD) in the same language (Arabizi) as that of the fine-tuning dataset (SALAD), outperforms a state-of-the-art multi-lingual pretrained model (multilingual BERT) on a sentiment analysis task.

Turn change prediction and next speaker prediction are two important tasks in multimodal, multiparty human-agent interaction. Predicting a change of dialogue turn and the most probable next speaker can help an agent to decide whether he should contribute to the discussion or wait for someone else to speak. In this research, we propose a machine learning-based approach for both turn change and next speaker prediction. Individual as well as combined models are explored to tackle these tasks. Results show that the proposed models outperform baselines. An ablation study is also performed to measure the importance of different features.

Addressee detection is an important challenge to tackle in order to improve dialogical interactions between humans and agents. This detection, essential for turn-taking models, is a hard task in multiparty conditions. Rule based as well as statistical approaches have been explored. Statistical approaches, particularly deep learning approaches, require a huge amount of data to train. However, smart feature selection can help improve addressee detection on small datasets, particularly if multimodal information is available. In this article, we propose a statistical approach based on smart feature selection that exploits contextual and multimodal information for addressee detection. The results show that our model outperforms an existing baseline.

Addressee detection is one of the most fundamental tasks for seamless dialogue management and turn taking in human-agent interaction. Whereas addressee detection is implicit in dyadic interaction, it becomes a challenging task in multiparty interactions when more than two participants are involved. Existing research works employ either rule-based or statistical approaches for addressee detection. However, most of these works either have been tested on a single data set or only support a fixed number of participants. In this article, we propose a model based on generic features to predict the addressee in data sets with varying number of participants. The results tested on two different corpora show that the proposed model outperforms existing baselines.

Automatic dialogue act annotation of speech utterances is an important task in human-agent interaction in order to correctly interpret user utterances. Speech utterances can be transcribed manually or via Automatic Speech Recognizer (ASR). In this article, several Machine Learning models are trained on manual and ASR transcriptions of user utterances, using bag of words and n-grams feature generation approaches, and evaluated on ASR transcribed test set. Results show that models trained using ASR transcriptions perform better than algorithms trained on manual transcription. The impact of irregular distribution of dialogue acts on the accuracy of statistical models is also investigated, and a partial solution to this issue is shown using multimodal information as input.

Intelligent Virtual Agents are suitable means for interactive storytelling for children. The engagement level of child interaction with virtual agents is a challenging issue in this area. However, the characteristics of child-agent interaction received moderate to little attention in scientific studies whereas such knowledge may be crucial to design specific applications.

Nowadays, Online Social Networks (OSN) are commonly used by groups of users to communicate. Members of a family, colleagues, fans of a brand, political groups... There is an increasing demand for a precise identification of these groups, coming from brand monitoring, business intelligence and e-reputation management. However, a gap can be observed between the communities detected by many data analytics algorithms on OSN, and effective groups existing in real life: the detected communities often lack of meaning and internal semantic cohesion. Most of existing literature on OSN either focuses on the community detection problem in graphs without considering the topic of the messages exchanged, or concentrates exclusively on the messages without taking into account the social links. In this article, we support the hypothesis that communities extracted on OSN should be topically coherent. We therefore propose a model to represent the groups of interaction on Twitter, the reference on micro-blogging OSN, and two metrics to evaluate the topical cohesion of the detected communities. As an evaluation, we measure the topical cohesion of the groups of users detected by a baseline community detection algorithm.

Nowadays, Online Social Networks (OSN) are commonly used by groups of users to communicate. Members of a family, colleagues, fans of a brand, political groups: the demand for a precise identification of these groups is increasing from brand monitoring, business intelligence and e-reputation management.

Our work aims at designing a dialogue manager dedicated to agents that interact with humans. In this article, we show how empirically specified dialogue games can be employed on both interpretative and generative levels of dialogue management. We present DOGMA, an open-source module that can be used by an agent to manage its conventional communicative behaviour. We show that our library of dialogue games can be used into DOGMA to generate fragments of dialogue that are strongly coherent from a human perspective.

Les systèmes interactifs incarnés, virtuels ou robotiques, sont de plus en plus riches en terme de capacités dialogiques et affectives, ainsi que de comportements multimodaux.Cependant, cette richesse rend difficile la conception de nouveaux systèmes.Dans ce cadre, une approche par composants permet d'une part de réutiliser ceux-ci dans différents contextes et d'autre part, de répartir les traitements inhérents à l'interaction.Dans cet article, une démonstration des nouvelles fonctionnalités de la plateforme AgentSlang est présentée.Les améliorations apportées sont principalement de deux ordres : (1) l'intégration de mécanismes de dialogue fondés sur des modèles issus de l'apprentissage automatique, et (2) le support des agents robotiques.Ces ajouts permettent d'améliorer le potentiel de prototypage rapide d'AgentSlang, notamment pour étudier un modèle d'interaction au travers de plusieurs types d'incarnations.

This paper addresses the problem of feature fusion between smile, as a visual feature, and text, as a transcription result. The influence of smile over semantic data has been considered before, without investigating multiple approaches for the fusion. This problem is multi-modal, which makes it more difficult. The goal of this article is to investigate how this fusion could increase the current interactivity of a dialogue system by boosting the automatic detection rate of the sentiments expressed by a human user. There are two original propositions in our approach. The first lies in the use of a segmented detection for text data, rather than predicting a single label for every document (video). Second, this paper studies the importance of several features in the process of multi-modal fusion. Our approach uses basic features, such as NGrams, Smile Presence or Valence to find the best fusion approach. Moreover, we test a two level classification approach, using a SVM.

Sentiment analysis and affect detection algorithms are generally based on annotated data, structured into dictionaries, ontologies or word nets. Among other research problems, two issues are considered very important in this field: 1) word sense disambiguation and 2) accuracy of affect detection. Most of the current approaches use annotated resources based on word nets. Their structure, founded on synonymic relations, makes the disambiguation process very difficult. Our model uses contextonyms, which simplify the decision process. Therefore, the disambiguation issue is transformed into a context matching problem. The second focus is on the manual annotation of the data followed by a semantic valence propagation. This approach enables the generation of new affective labels from a set of initial ones, through the expansion process. Unfortunately, this is usually done to the detriment of precision. We use an existing linguistic resource, SentiWordNet, which is one of the largest dictionaries available for sentiment analysis. Using our disambiguation model, we manage to solve all the SentiWordNet ambiguities and inconsistencies, which increases the accuracy of the classification process. This is the first of our major contributions. Second, we manage to reduce the disagreement percentage computed against well known linguistic resources to less than half of the original rate.