The ICMI 2025 Workshop on Holistic and Responsible Affective Intelligence (HRAI 2025) aims to advance research in affective intelligence by fostering discussions on the holistic development of affective computing and the ethical challenges it entails. The workshop aims to strengthen interdisciplinary connections within the affective computing community, promoting better integration of methodologies and enhancing real-world applicability. By tackling both technical and ethical issues, HRAI 2025 aspires to shape the future of affective AI, ensuring it is not only powerful but also fair, safe, and socially responsible.

In this paper we propose to use Recurrence Plots (RP) to generate 2D representations of physiological activity which should be less subject dependent and better suited for non-stationary signals such as EDA. The performance of spectrograms and RPs are compared on two publicly available datasets: AMIGOS and DEAP. Transfer learning is employed by using a pre-trained ResNet-50 model to recognize emotional states (high vs low arousal and high vs low valence) from the two types of representations. Results show that RPs reach a similar performance to spectrograms on periodic signals such as ECG and plethysmography (F1 of 0.76 for valence and 0.74 for arousal on the AMIGOS dataset) while they outperform spectrograms on EDA (F1 of 0.74 for valence and 0.75 for arousal). By combining the two sources of information we were able to reach a F1 of 0.76 for valence and 0.75 for arousal.

We present a dataset (IMPRESSION) for multi-modal recognition of impressions on individuals and dyads. Compared to other databases, we did not only elicit impression using video stimuli, but also recorded natural impression formation of strangers meeting for the first time through video call. The database allows machine learning studies on impression recognition, using multimodal signals of individuals in relation to their emotion expressivity, and with respect to the interlocutor’s reactions. The experiment setup was arranged with 62 participants’ synchronized recordings of face videos, audio signals, eye gaze data, and peripheral nervous system physiological signals (Electrocardiogram-ECG, Blood Volume Pulse-BVP and Galvanic Skin Response-GSR) using wearable sensors. Participants reported their formed impressions in the W & C dimensions in real-time. We present the database in detail as well as baseline methods and results for impression recognition in W & C.

Recognizing emotional reactions of movie audiences to affective movie content is a challenging task in affective computing. Previous research on induced emotion recognition has mainly focused on using audio-visual movie content. Nevertheless, the relationship between the perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audiences (induced emotions) is unexplored. In this work, we studied the relationship between perceived and induced emotions of movie audiences. Moreover, we investigated multimodal modelling approaches to predict movie induced emotions from movie content based features, as well as physiological and behavioral reactions of movie audiences. To carry out analysis of induced and perceived emotions, we first extended an existing database for movie affect analysis by annotating perceived emotions in a crowd-sourced manner. We find that perceived and induced emotions are not always consistent with each other. In addition, we show that perceived emotions, movie dialogues, and aesthetic highlights are discriminative for movie induced emotion recognition besides spectators' physiological and behavioral reactions. Also, our experiments revealed that induced emotion recognition could benefit from including temporal information and performing multimodal fusion. Moreover, our work deeply investigated the gap between affective content analysis and induced emotion recognition by gaining insight into the relationships between aesthetic highlights, induced emotions, and perceived emotions.

Horror games represent a very specific genre specifically designed to elicit fear. These games provide a tremendous emotional experience balanced between stress and satisfaction. Yet, over time, the player acquires further insight of the mechanisms of the game, dissipating the creepy climate that reduces the emotional impact intended. This article hypothesises that exploring existing facets within a game such as visuals and sounds might establish a good approach to renew the gaming experience. To understand the players' emotional reactions toward context alteration, an adaptation of a published game (P.T. by Konami 2014) was used. This context refers mainly to light effects, sounds, and in-game events. To learn which game effects induce the strongest physiological reactions, an experiment was conducted, and correlation between the physiological data, collected through the measure of the galvanic skin response, and that of the perceived emotion provided by participants, was investigated. Results show that the order in which effects are arranged can produce extensive emotional responses. They also suggest that psychological impact can be increased not only by the visual horror itself, but also through the process that slowly builds up to it, in particular the usage of sounds.

This paper presents a computational model for managing an Embodied Conversational Agent's first impressions of warmth and competence towards the user. These impressions are important to manage because they can impact users' perception of the agent and their willingness to continue the interaction with the agent. The model aims at detecting user's impression of the agent and producing appropriate agent's verbal and nonverbal behaviours in order to maintain a positive impression of warmth and competence. User's impressions are recognized using a machine learning approach with facial expressions (action units) which are important indicators of users' affective states and intentions. The agent adapts in real-time its verbal and nonverbal behaviour, with a reinforcement learning algorithm that takes user's impressions as reward to select the most appropriate combination of verbal and non-verbal behaviour to perform. A user study to test the model in a contextualized interaction with users is also presented. Our hypotheses are that users' ratings differs when the agents adapts its behaviour according to our reinforcement learning algorithm, compared to when the agent does not adapt its behaviour to user's reactions (i.e., when it randomly selects its behaviours). The study shows a general tendency for the agent to perform better when using our model than in the random condition. Significant results shows that user's ratings about agent's warmth are influenced by their a-priori about virtual characters, as well as that users' judged the agent as more competent when it adapted its behaviour compared to random condition.

Detection of aesthetic highlights is a challenge for understanding the affective processes taking place during movie watching. In this article, we study spectators’ responses to movie aesthetic stimuli in a social context. Moreover, we look for uncovering the emotional component of aesthetic highlights in movies. Our assumption is that synchronized spectators’ physiological and behavioral reactions occur during these highlights because: ( i ) aesthetic choices of filmmakers are made to elicit specific emotional reactions (e.g., special effects, empathy, and compassion toward a character) and ( ii ) watching a movie together causes spectators’ affective reactions to be synchronized through emotional contagion. We compare different approaches to estimation of synchronization among multiple spectators’ signals, such as pairwise, group, and overall synchronization measures to detect aesthetic highlights in movies. The results show that the unsupervised architecture relying on synchronization measures is able to capture different properties of spectators’ synchronization and detect aesthetic highlights based on both spectators’ electrodermal and acceleration signals. We discover that pairwise synchronization measures perform the most accurately independently of the category of the highlights and movie genres. Moreover, we observe that electrodermal signals have more discriminative power than acceleration signals for highlight detection.

This paper studies emotion recognition in the context of collaboration. When people are interacting with each other they tend to reach a similar emotional state through mechanisms like empathy and emotion contagion. We thus investigated if participants' emotions could be determined from the affective reactions and behaviors of their partner. Two types of emotional expressions were studied: physiological reactions and speech. Results show that emotions could be recognized with similar performance when employing affective features from the self or the partner. In addition, performance was improved when combining self and partner information. The results demonstrate that in social situations an emotion recognition model should include information about partners.

Predicting the emotional response of movie audiences to affective movie content is a challenging task in affective computing. Previous work has focused on using audiovisual movie content to predict movie induced emotions. However, the relationship between the audience's perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audience (induced emotions) remains unexplored. In this work, we address the relationship between perceived and induced emotions in movies, and identify features and modelling approaches effective for predicting movie induced emotions. First, we extend the LIRIS-ACCEDE database by annotating perceived emotions in a crowd-sourced manner, and find that perceived and induced emotions are not always consistent. Second, we show that dialogue events and aesthetic highlights are effective predictors of movie induced emotions. In addition to movie based features, we also study physiological and behavioural measurements of audiences. Our experiments show that induced emotion recognition can benefit from including temporal context and from including multimodal information. Our study bridges the gap between affective content analysis and induced emotion prediction.

The papers in this special section focus on commercial applications for affective computing. One of the main goals of affective computing is to create machines which can adapt to users’ emotions in order to produce more natural and efficient interaction. Emotion recognition is thus a central component of the field, and is based on a variety of measurements (facial expressions, speech, gait patterns, physiology, eye tracking, etc.) that are analyzed using advanced pattern recognition techniques. Furthermore, researchers and entrepreneurs have identified countless possible applications of affect-aware technology, from health and driver monitoring to exercise [6] and computer game adaptation. However, although great scientific advances have been made and many applications have been proposed, only few robust implementations have been presented or validated, and commercial adoption of affect-aware technology has been marginal. One rare example of a successful application is the smile detector used in digital cameras to automatically take pictures when the subject is smiling. This weak adoption of the technology can be attributed to several unsolved challenges in the domain of affective computing.

Detection of emotional and aesthetic highlights is a challenge for the affective understanding of movies. Our assumption is that synchronized spectators' physiological and behavioral reactions occur during these highlights. We propose to employ the periodicity score to capture synchronization among groups of spectators' signals. To uncover the periodicity score's capabilities, we compare it with baseline synchronization measures, such as the nonlinear interdependence and the windowed mutual information. The results show that the periodicity score and the pairwise synchronization measures are able to capture different properties of spectators' synchronization, and they indicate the presence of some types of emotional and aesthetic highlights in a movie based on spectators' electro-dermal and acceleration signals.

Detection of highlights in movies is a challenge for the affective understanding and implicit tagging of films. Under the hypothesis that synchronization of the reaction of spectators indicates such highlights, we define a synchronization measure between spectators that is capable of extracting movie highlights. The intuitive idea of our approach is to define (a) a parameterization of one spectator's physiological data on a manifold; (b) the synchronization measure between spectators as the Kolmogorov-Smirnov distance between local shape distributions of the underlying manifolds. We evaluate our approach using data collected in an experiment where the electro-dermal activity of spectators was recorded during the entire projection of a movie in a cinema. We compare our methodology with baseline synchronization measures, such as correlation, Spearman's rank correlation, mutual information, Kolmogorov-Smirnov distance. Results indicate that the proposed approach allows to accurately distinguish highlight from non-highlight scenes.

Physiological and affective computing propose methods to improve human–machine interactions by adapting machines to the users’ states. Recently, social signal processing (SSP) has proposed to apply similar methods to human–human interactions with the hope of better understanding and modeling social interactions. Most of the social signals employed are facial expressions, body movements and speech, but studies using physiological signals remain scarce. In this paper, we motivate the use of physiological signals in the context of social interactions. Specifically, we review studies which have investigated the relationship between various physiological indices and social interactions. We then propose two main directions to apply physiological SSP: using physiological signals of individual users as new social cues displayed in the group and using inter-user physiology to measure properties of the interactions such as conflict and social presence. We conclude that physiological measures have the potential to enhance social interactions and to connect people.

In this paper we propose a method to assess key collaborative processes during computer-supported group work based on physiological signals and eye-movements. Synchronous interpersonal multimodal signals from 30 dyads were recorded while collaborating remotely. Features measuring how much collaborators' eye-movements and physiology are coupled were extracted from the obtained time series and two regression models were trained to assess collaboration. Results show that the two coupling measures can be used to predict collaborative processes such as grounding and convergence. Assessing those processes is a major step toward the development of remote collaborative interfaces able to adapt to the users' social interactions.

Following the first and second workshop on affective brain-computer interfaces, held in conjunction with ACII in Amsterdam (2009) and Memphis (2011), the third workshop explores the advantages and limitations of using neurophysiological signals for the automatic recognition of affective and cognitive states, and the different ways to use this information about the user in applications within the health, arts, and entertainment domains. The goal is to bring researchers, artists, and practitioners together to present state-of-the-art progress, discuss pitfalls and limitations and share and create visions, and thereby encourage the development of guidelines and frameworks for affective BCI.

The proposed demonstration is an automatic emotion assessment installation used for game's dynamic difficulty adjustment. The goal of the system is to maintain the player of the game in a state of entertainment and engagement where his/her skills match the difficulty level of the game. The player's physiological signals are recorded while playing a Tetris game and signal processing, feature extraction and classification techniques are applied to the signals in order to detect when the player is anxious or bored. The level of the Tetris game is then adjusted according to the player's detected emotional state. The demonstration will also serve as an experimental protocol to test the player's experience through their interaction with the proposed platform.

We report the results of an empirical study demonstrating the value of using physiological compliance as a measure of social presence during digital game playing. The physiological activity (facial EMG, electrodermal activity, cardiac activity and respiration) of 21 dyads were acquired synchronously while they were playing a digital game either cooperatively or competitively and either at home or in the laboratory. Physiological compliance was defined as the correlation between the physiological signals of the dyad members. The results of this study confirm that physiological compliance is higher in a conflicting situation than when playing cooperatively. Importantly, the results also demonstrate that physiological compliance is related to self-reported social presence. This suggests that physiological compliance is not limited to negative situations but rather increases due to rich interactions. Only minor differences in physiological compliance were observed between home play and laboratory play, suggesting the ecological validity of laboratory measures. Finally, we propose that compliance measures can be considered as objective indices of social presence in digital gaming.

This paper proposes to maintain player's engagement by adapting game difficulty according to player's emotions assessed from physiological signals. The validity of this approach was first tested by analyzing the questionnaire responses, electroencephalogram (EEG) signals, and peripheral signals of the players playing a Tetris game at three difficulty levels. This analysis confirms that the different difficulty levels correspond to distinguishable emotions, and that, playing several times at the same difficulty level gives rise to boredom. The next step was to train several classifiers to automatically detect the three emotional classes from EEG and peripheral signals in a player-independent framework. By using either type of signals, the emotional classes were successfully recovered, with EEG having a better accuracy than peripheral signals on short periods of time. After the fusion of the two signal categories, the accuracy raised up to 63%.

In this paper, we propose an approach for affective representation of movie scenes based on the emotions that are actually felt by spectators. Such a representation can be used for characterizing the emotional content of video clips for e.g. affective video indexing and retrieval, neuromarketing studies, etc. A dataset of 64 different scenes from eight movies was shown to eight participants. While watching these clips, their physiological responses were recorded. The participants were also asked to self-assess their felt emotional arousal and valence for each scene. In addition, content-based audio- and video-based features were extracted from the movie scenes in order to characterize each one. Degrees of arousal and valence were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based features. We showed that a significant correlation exists between arousal/valence provided by the spectator's self-assessments, and affective grades obtained automatically from either physiological responses or from audio-video features. This demonstrates the ability of using multimedia features and physiological responses to predict the expected affect of the user in response to the emotional video content.

The work presented in this paper aims at assessing human emotions using peripheral as well as electroencephalographic (EEG) physiological signals. Three specific areas of the valence-arousal emotional space are defined, corresponding to negatively excited, positively excited, and calm-neutral states. An acquisition protocol based on the recall of past emotional events has been designed to acquire data from both peripheral and EEG signals. Pattern classification is used to distinguish between the three areas of the valence-arousal space. The performance of two classifiers has been evaluated on different features sets: peripheral data, EEG data, and EEG data with prior feature selection. Comparison of results obtained using either peripheral or EEG signals confirms the interest of using EEG’s to assess valence and arousal in emotion recall conditions.