The ICMI 2025 Workshop on Holistic and Responsible Affective Intelligence (HRAI 2025) aims to advance research in affective intelligence by fostering discussions on the holistic development of affective computing and the ethical challenges it entails. The workshop aims to strengthen interdisciplinary connections within the affective computing community, promoting better integration of methodologies and enhancing real-world applicability. By tackling both technical and ethical issues, HRAI 2025 aspires to shape the future of affective AI, ensuring it is not only powerful but also fair, safe, and socially responsible.

High-dosage tutoring has become an effective strategy for bolstering K-12 academic performance and combating education declines accelerated by the COVID-19 pandemic. To achieve high-dosage tutoring at scale, tutoring programs often rely on paraprofessional tutors—recruited tutors with college degrees who lack formal training in education—however, these tutors may require consistent and targeted feedback from instructional coaches for improvement. Accordingly, we developed a human-tutor coaching technology (HTCT) system to automatically extract discourse analytics pertaining to accountable talk moves (or academically productive talk) from tutoring sessions and provide feedback visualizations to coaches to aid their coaching sessions with tutors. We deployed HTCT in a user study using a virtual tutoring platform with 11 real coaches, 40 tutors, and their students to investigate coaches’ usage patterns with HTCT, perceptions of its utility, and changes in tutors’ talk. Overall, we found that coaches had positive perceptions of the system. We also observed an increase in accountable talk from tutors whose coaches used HTCT compared to tutors whose coaches did not. We discuss implications for AI-based applications which offer coaches a promising way to provide personalized, automated, and data-driven feedback to scale high-dosage tutoring.

Shared visual attention between team members facilitates collaborative problem solving (CPS), but little is known about how team-level eye gaze dynamics influence the quality and successfulness of CPS. To better understand the role of shared visual attention during CPS, we collected eye gaze data from 279 individuals solving computer-based physics puzzles while in teams of three. We converted eye gaze into discrete screen locations and quantified team-level gaze dynamics using recurrence quantification analysis (RQA). Specifically, we used a centroid-based auto-RQA approach, a pairwise team member cross-RQAs approach, and a multi-dimensional RQA approach to quantify team-level eye gaze dynamics from the eye gaze data of team members. We find that teams differing in composition based on prior task knowledge, gender, and race show few differences in team-level eye gaze dynamics. We also find that RQA metrics of team-level eye gaze dynamics were predictive of task success (all ps < .001). However, the same metrics showed different patterns of feature importance depending on predictive model and RQA type, suggesting some redundancy in task-relevant information. These findings signify that team-level eye gaze dynamics play an important role in CPS and that different forms of RQA pick up on unique aspects of shared attention between team-members.

Engagement is critical to satisfaction and performance in a number of domains but is challenging to measure and sustain. Thus, there is considerable interest in developing affective computing technologies to automatically measure and enhance engagement, especially in the wild and at scale. This article provides an accessible introduction to affective computing research on engagement detection and enhancement using educational applications as an application domain. We begin with defining engagement as a multicomponential construct (i.e., a conceptual entity) situated within a context and bounded by time and review how the past six years of research has conceptualized it. Next, we examine traditional and affective computing methods for measuring engagement and discuss their relative strengths and limitations. Then, we move to a review of proactive and reactive approaches to enhancing engagement toward improving the learning experience and outcomes. We underscore key concerns in engagement measurement and enhancement, especially in digitally enhanced learning contexts, and conclude with several open questions and promising opportunities for future work.

Given the widespread adverse outcomes of stress – exacerbated by the current pandemic – wearable sensing provides unique opportunities for automated stress tracking to inform well-being interventions. However, its success in the wild and at scale depends on the robustness and validity of automated stress inference, which is limited in current systems. In this work, we enumerate the properties of robustness and validity necessary for achieving viable automated stress inference using wearable sensors, and we underscore present challenges to constructing and evaluating these systems. Using these criteria as guiding principles, we present automated stress inference results from a large (N=606)in situlongitudinal wearable and contextual sensing study of information workers. Using a multimodal approach encompassing a wearable sensor, relative location tracking, smartphone usage, and environmental sensing, we trained regression models to predict daily self-reported perceived stress in a participant-independent fashion. Our models significantly outperformed baseline variants with shuffled stress scores and were consistent with small-to-moderate effects. Our findings highlight the performance disparity between robust and valid approaches to automated perceived stress inference and current approaches and suggest that further performance gains might require additional sensing modalities and enhanced contextual awareness than existing approaches.

We introduce the psychometric concepts of bias and fairness in a multimodal machine learning context assessing individuals’ hireability from prerecorded video interviews. We collected interviews from 733 participants and hireability ratings from a panel of trained annotators in a simulated hiring study, and then trained interpretable machine learning models on verbal, paraverbal, and visual features extracted from the videos to investigate unimodal versus multimodal bias and fairness. Our results demonstrate that, in the absence of any bias mitigation strategy, combining multiple modalities only marginally improves prediction accuracy at the cost of increasing bias and reducing fairness compared to the least biased and most fair unimodal predictor set (verbal). We further show that gender-norming predictors only reduces gender predictability for paraverbal and visual modalities, while removing gender-biased features can achieve gender blindness, minimal bias, and fairness (for all modalities except for visual) at the cost of some prediction accuracy. Overall, the reduced-feature approach using predictors from all modalities achieved the best balance between accuracy, bias, and fairness, with the verbal modality alone performing almost as well. Our analysis highlights how optimizing model prediction accuracy in isolation and in a multimodal context may cause bias, disparate impact, and potential social harm, while a more holistic optimization approach based on accuracy, bias, and fairness can avoid these pitfalls.

Generating accurate ground truth representations of human subjective experiences and judgements is essential for advancing our understanding of human-centered constructs such as emotions. Often, this requires the collection and fusion of annotations from several people where each one is subject to valuation disagreements, distraction artifacts, and other error sources. This work proposes trapezoidal segment sequencing, a new method for fusing annotations into a single representation that, when used alongside a recently proposed signal warping pipeline for correcting annotation artifacts, produces accurate ground truths. We prove that annotations can be well approximated with trapezoidal signals and present results showing the proposed method performs competitively with state-of-the-art fusion methods on a data set where the true target signal being annotated is known. The main utility of the proposed approach is its ability to help segment individual annotations into interpretable regions where either changes or no perceived changes to the construct occur.

Wearable sensors today can unobtrusively collect rich time-series of physiological states and human movement patterns over a prolonged period. Gaining a better understanding of how an individual's physiological responses vary in different workplace environments can be valuable in understanding human behavior related to wellness and performance. In this work, we describe our exploration in discovering the correlation between one's physiological responses and movement patterns within different indoor locations using data collected from nurses in a hospital workplace for a ten week period. In this work, we use simple heuristics to empirically validate the idea that such a relationship may exist and then quantify it using mutual information analysis. We propose and demonstrate a data analysis approach that can also detect variations in the level of mutual dependency between different locations and physiological responses. The mutual dependency measures derived from our method are empirically shown to provide valuable information for improving modeling of self-reported work behavior patterns compared to using features derived from a single data stream.

Continuous human annotations of complex human experiences are essential for enabling psychological and machine-learned inquiry into the human mind, but establishing a reliable set of annotations for analysis and ground truth generation is difficult. Measures of consensus or agreement are often used to establish the reliability of a collection of annotations and thereby purport their suitability for further research and analysis. This work examines many of the commonly used agreement metrics for continuous-scale and continuous-time human annotations and demonstrates their shortcomings, especially in measuring agreement in general annotation shape and structure. Annotation quality is carefully examined in a controlled study where the true target signal is known and evidence is presented suggesting that annotators' perceptual distortions can be modeled using monotonic functions. A novel measure of agreement is proposed which is agnostic to these perceptual differences between annotators and provides unique information when assessing agreement. We illustrate how this measure complements existing agreement metrics and can serve as a tool for curating a reliable collection of human annotations based on differential consensus.

Accurate ground truth representations of human behavior and experiences are essential for furthering our understanding of the complex relationships between everyday events and interactions and their effects on people. Producing accurate ground truth signals for subjective or latent experiences is difficult because it requires human annotation and is subject to annotator bias, distraction artifacts, valuation errors, among others. We build on previous work aiming to produce highly accurate continuous-scale ground truth labels for human experiences which advocates using supplemental human observations to warp the continuous-scale annotations to correct these errors. We propose a new method, trapezoidal segmented regression, for optimally approximating fused human-produced continuous-scale annotations to simplify its segmentation into intervals of low and high confidence in valuation. We evaluate this algorithm as an alternative to the total variation denoising method used in prior work by comparing the ground truths that both methods produce in experiments where the true annotation target signal is known a priori. Results show that the proposed signal approximation technique performs on par with the prior method, producing ground truth signals in close alignment with the true target, but with the added advantages of being more easily tuned and intuitive. We conclude that the proposed algorithm enables accurate and more robust ground truth generation.

Gaining a better understanding of how people move about and interact with their environment is an important piece of understanding human behavior. Careful analysis of individuals' deviations or variations in movement over time can provide an awareness about changes to their physical or mental state and may be helpful in tracking performance and well-being especially in workplace settings. We propose a technique for clustering and discovering patterns in human movement data by extracting motifs from the time series of durations where participants linger at different locations. Using a data set of over 200 participants moving around a hospital for ten weeks, we show this technique intuitively captures local temporal relationships between hospital rooms and also clusters them in a fashion consistent with the room type labels (e.g. lounge, break room, etc.) without using prior knowledge. Machine learning features derived from these clusters are empirically shown to provide information similar to features attained using domain knowledge of the room type labels directly when predicting mental wellness from self-reports.

Human annotations are of integral value in human behavior studies and in particular for the generation of ground truth for behavior prediction using various machine learning methods. These often subjective human annotations are especially required for studies involving measuring and predicting hidden mental states (e.g. emotions) that cannot effectively be measured or assessed by other means. Human annotations are noisy and prone to the influence of several factors including personal bias, task ambiguity, environmental distractions, and health state. We propose a novel method for fusion of continuous real-time human annotations to generate accurate ground truth estimates. We introduce a signal warping method that uses additional comparative rank-based information about specific subsets of the annotations to correct for specific types of human annotation artifacts. This approach is validated using a mechanically simple but perceptually demanding psychophysical annotation experiment where objective truth labels are known. Our method yields ground truth estimates that are in better agreement with the objective truth than state-of-the-art approaches.

Student behavior and lecturer oversight in the classroom is known to modulate study behaviors and impact performance and learning outcomes, but cannot at present be managed for distance learning students. Quantifying and automatically measuring student engagement during lectures in a scalable and accessible manner for these students is essential for improving academic success, but has not been studied widely in natural distance learning environments. We collect video recordings from a screen-mounted camera of students studying online lectures in a mostly unstructured setting and gather annotations from a panel of humans for assessing student engagement levels. We present results on the prediction of different representations of engagement, both with subject-independent and individual-specific models, and quantify the performance gap between the generalized and personalized models for engagement prediction. While the subject-independent performance is challenged by data sparsity, results show that the individual-specific models can predict engagement well even with very few labeled examples.