Cohesion is a multidimensional construct reflecting a shared commitment to group task(s) and strong interpersonal bonds.It plays a crucial role in collective success and emotional well-being.In human-computer interactions, embodied conversational agents can help foster group cohesion and prevent negative group dynamics by detecting and predicting low cohesion in real-time.This study introduces a novel social signal processing framework for real-time cohesion estimation in hybrid group interactions and develops computational models based on it.Using a multiparty interaction corpus, we extract nonverbal social signals available in real-time and focus on estimating overall cohesion and its dimensions (social, task).We further examine the contribution of social signals to the accuracy of low cohesion detection in collaborative interactions.In our experimental setting, overall low cohesion is more effectively detected than its social or task dimensions.The ablation study suggests that specific social signals reflecting interindividual interaction patterns substantially contribute to low cohesion detection performance.This work demonstrates the potential of data-driven methods for real-time cohesion estimation in multiparty interactions, highlighting promising opportunities for ECAs to restore and maintain cohesion in hybrid group settings.

Quality assurance in manufacturing companies is an essential process for ensuring that products meet established standards. It contributes to customer satisfaction, as well as the reduction of the costs associated with defects. With Quality 4.0, an extension of Industry 4.0 to quality assurance, new possibilities in terms of product quality management are emerging. Thanks to expert knowledge, data collected by machine sensors can be used to anticipate quality issues or manufacturing errors. To semantically detect those situations, an ontology representing manufacturing knowledge linked to quality detection situations is needed. Moreover, as heterogeneous data streams have to be integrated, a combination of stream processing and of-line reasoning can be used. This combination allows a continuous process of data and the use of expert knowledge to detect anomalies. This paper presents an approach for detecting manufacturing quality losses. Therefore, an ontology-based context for manufacturing is introduced to detect quality issues situations. Then, an extension of an existing model using stream reasoning to process heterogeneous data from sensors and predictions is presented to detect the situations continuously.

In order to improve multi-party social interaction with artificial companions such as robots or virtual agents, real-time Emotion Recognition in Conversation (ERC) is required. In this context, ERC is a challenging task which involves multiple challenges, such as processing multimodal data over time, taking into account the multi-party context with any number of participants, understanding implied relevant commonsense knowledge during interaction and taking into account each participant’s emotional attitude. To deal with the aforementioned challenges, we design a multimodal off-the-shelf model that meets the requirements of real-life scenarios, specifically dyadic and multi-party interactions. We propose a Knowledge Aware Multi-Headed Network that integrates various sources including the dialog history and commonsense knowledge about the speaker and other participants. The weights of these pieces of information are modulated using a multi-head attention mechanism. The proposed model is learnt in a Multi-Task Learning framework which combines the ERC task with a Dialogue Act (DA) recognition task and an Emotion Shift (ES) detection task through a joint learning strategy. Our proposition obtains competitive and stable results on several benchmark datasets that vary in number of participants and length of conversations, and outperforms the state-of-the-art on one of these datasets. The importance of DA and ES prediction in determining the speaker’s current emotional state is investigated.

Empathic behavior between humans often has a positive effect, particularly in healthcare, since it facilitates relationships, improves engagement, and reduces stress and anxiety. Despite the importance of empathic communication and social relationship in healthcare, effects of empathic behavior of embodied virtual agents that interact with patients in a multimodal and adaptive way have not been widely explored. In this article, we propose an empathic model which endows a therapeutic embodied virtual agent with multi-modal adaptive empathic behavior during interaction with a user. This model relies on the user-agent interaction relationship and focuses on (1) the interpretation of user's behavior using multimodal input, and (2) the generation of multimodal empathic behavior during interaction. An experimental study in the context of empathic interaction with students during the COVID-19 pandemic is presented to evaluate the effects of adaptive empathic behavior of an agent on the quality of user interaction. Compared to an agent that relies on low-level affect matching and backchannels, results show that our agent is perceived as more empathic and improves user engagement during the interaction.

Empathic behavior between humans often has a positive effect, particularly in healthcare, since it facilitates relationship, improves engagement, and reduces stress and anxiety. Despite this importance of empathic communication and social relationship in healthcare, the effects of empathic behavior of embodied virtual agents that interact with patients in a multimodal and adaptive way have not been widely explored. The proposed multimodal adaptive empathic agent architecture (MAEAA) endows a therapeutic embodied virtual agent (EVA) with adaptive empathic behavior during interaction with a user. This architecture relies on user-agent interaction relationship and focuses on (1) the interpretation of user's behavior using multimodal input, and (2) the generation of multimodal empathic behavior during interaction. The solution have been used to develop a user study for the empathic interaction with students in the healthcare context [3].

Turn change prediction and next speaker prediction are two important tasks in multimodal, multiparty human-agent interaction. Predicting a change of dialogue turn and the most probable next speaker can help an agent to decide whether he should contribute to the discussion or wait for someone else to speak. In this research, we propose a machine learning-based approach for both turn change and next speaker prediction. Individual as well as combined models are explored to tackle these tasks. Results show that the proposed models outperform baselines. An ablation study is also performed to measure the importance of different features.

Addressee detection is an important challenge to tackle in order to improve dialogical interactions between humans and agents. This detection, essential for turn-taking models, is a hard task in multiparty conditions. Rule based as well as statistical approaches have been explored. Statistical approaches, particularly deep learning approaches, require a huge amount of data to train. However, smart feature selection can help improve addressee detection on small datasets, particularly if multimodal information is available. In this article, we propose a statistical approach based on smart feature selection that exploits contextual and multimodal information for addressee detection. The results show that our model outperforms an existing baseline.

Addressee detection is one of the most fundamental tasks for seamless dialogue management and turn taking in human-agent interaction. Whereas addressee detection is implicit in dyadic interaction, it becomes a challenging task in multiparty interactions when more than two participants are involved. Existing research works employ either rule-based or statistical approaches for addressee detection. However, most of these works either have been tested on a single data set or only support a fixed number of participants. In this article, we propose a model based on generic features to predict the addressee in data sets with varying number of participants. The results tested on two different corpora show that the proposed model outperforms existing baselines.

With the coming era of Industry 4.0, more assets and machines in plants are equipped with sensors which collect big amount of data for effective on-line equipment condition monitoring. Monitoring equipment conditions can not only reduce unplanned downtime by early detection of relevant situations like anomalies but also avoid unnecessary routine maintenance. For the detection of these situations it is necessary to integrate distributed, heterogeneous data sources and data streams. In this context, semantic web technologies are increasingly considered as key technologies to improve data integration. However, they are mainly used for data that is assumed not to change very often in time. In order to tackle this issue, stream reasoning combines reasoning and stream processing methods. Such a combination enables the processing of dynamic and heterogeneous data continuously produced from a large number of sources and implementing real-time services. This paper presents an approach that uses stream reasoning to identify in real time certain situations that lead to potential failures. Early detection enables to choose the most appropriate decision to avoid the interruption of manufacturing processes. In order to achieve this, data collected from sensors are enriched with contextual information. The use of stream reasoning allows the integration of data from different data sources, with different underlying meanings, different temporal resolutions as well as the processing of these data in real time.

Automatic dialogue act annotation of speech utterances is an important task in human-agent interaction in order to correctly interpret user utterances. Speech utterances can be transcribed manually or via Automatic Speech Recognizer (ASR). In this article, several Machine Learning models are trained on manual and ASR transcriptions of user utterances, using bag of words and n-grams feature generation approaches, and evaluated on ASR transcribed test set. Results show that models trained using ASR transcriptions perform better than algorithms trained on manual transcription. The impact of irregular distribution of dialogue acts on the accuracy of statistical models is also investigated, and a partial solution to this issue is shown using multimodal information as input.

Industry 4.0 is an initiative combining a set of technologies that help to achieve more efficient manufacturing processes. An important characteristic for industrial production in Industry 4.0 is that physical items such as sensors, devices and enterprise assets are connected to each other and to the Internet. In this environment, devices and sensors generate increasing amount of data. A key point to consider is that the execution of industrial processes should depend not only on their internal state and on user interactions but also on the context of their execution, in order to become context-aware and provide added-value information to improve the monitoring of operations and their performance. Ontologies emerge as a relevant method for representing manufacturing knowledge in a machine-interpretable way. Therefore, an ontology-based context model for industry is introduced in this paper. The model facilitates context representation and reasoning by providing structures for context-related concepts, rules and their semantics.

Nowadays, social media sites are growing day after day and they are gathering the most important number of visitors on the internet. But among these visitors there is also a big number of manipulators that try to benefit from the others for many reasons, which leads to increase the need to detect t he fake accounts that try to manipulate over the online social media (OSN). A malicious account on OSNs can be created for different objectives such as social spam, identity theft, spear phishing and sybil attacks. In this article, we study and analyze the multiple fake accounts created by one user in order to manipulate and bypass the OSN regulation. We present a new methodology to detect the multiple identity fake accounts and validate it on a set of 10000 accounts extracted from English Wikipedia (EnWiki). In this methodology, we propose a set of features that differentiate between fake and legitimate accounts and grows on previous literature, then we train and test them using machine learning algorithms. The results compare several machine learning algorithms to show that our new features and training data enable to detect 99% of fake accounts, improving previous results from the literature.

This paper presents a method for evaluating the credibility of agents’ behaviors in immersive multi-agent simulations. It combines two approaches. The first one is based on a qualitative analysis of questionnaires filled by the users and annotations filled by others participants to draw categories of users (related to their behavior in the context of the simulation or in real life). The second one carries out a quantitative behavior data collection during simulations in order to automatically extract behavior clusters. We then study the similarities between user categories, participants’ annotations and behavior clusters. Afterward, relying on user categories and annotations, we compare human behaviors to agent ones in order to evaluate the agents’ credibility and make their behaviors explicit. We illustrate our method with an immersive driving simulator experiment.