Empathy is increasingly recognized as a key factor in human–AI communication, yet conventional approaches to “digital empathy” often focus on simulating internal, human like emotional states while overlooking the inherently subjective, contextual, and relational facets of empathy as perceived by users. In this work, we propose a human-centered taxonomy that emphasizes observable empathic behaviors and introduce a new dataset, SENSE-7, of real-world conversations between information workers and Large Language Models (LLMs), which includes per-turn empathy annotations directly from the users, along with user characteristics, and contextual details, offering a more user-grounded representation of empathy. Analysis of 695 conversations from 109 participants reveals that empathy judgments are highly individualized, context-sensitive, and vulnerable to disruption when conversational continuity fails or user expectations go unmet. To promote further research, we provide a subset of 672 anonymized conversation and provide exploratory classification analysis, showing that an LLM-based classifier can recognize 5 levels of empathy with an encouraging average Spearman ρ = 0.369 and Accuracy = 0.487 over this set. Overall, our findings underscore the need for AI designs that dynamically tailor empathic behaviors to user contexts and goals, offering a roadmap for future research and practical development of socially attuned, human-centered artificial agents.

Client-Service Representatives (CSRs) are vital to organizations.Frequent interactions with disgruntled clients, however, disrupt their mental well-being.To help CSRs regulate their emotions while interacting with uncivil clients, we designed Care-Pilot, an LLM-powered assistant, and evaluated its efficacy, perception, and use.Our comparative analyses between 665 human and Care-Pilotgenerated support messages highlight Care-Pilot's ability to adapt to and demonstrate empathy in various incivility incidents.Additionally, 143 CSRs assessed Care-Pilot's empathy as more sincere and actionable than human messages.Finally, we interviewed 20 CSRs who interacted with Care-Pilot in a simulation exercise.They reported that Care-Pilot helped them avoid negative thinking, recenter thoughts, and humanize clients; showing potential for bridging gaps in coworker support.Yet, they also noted deployment challenges and emphasized the indispensability of shared experiences.We discuss future designs and societal implications of AI-mediated emotional labor, underscoring empathy as a critical function for AI assistants for worker mental health.

CHI ’25, Yokohama, Japan

Information workers increasingly struggle with productivity challenges in modern workplaces, facing difficulties in managing time and effectively utilizing workplace analytics data for behavioral improvement. Despite the availability of productivity metrics through enterprise tools, workers often fail to translate this data into actionable insights. We present a comprehensive, user-centric approach to address these challenges through AI-based productivity agents tailored to users' needs. Utilizing a two-phase method, we first conducted a survey with 363 participants, exploring various aspects of productivity, communication style, agent approach, personality traits, personalization, and privacy. Drawing on the survey insights, we developed a GPT-4 powered personalized productivity agent that utilizes telemetry data gathered via Viva Insights from information workers to provide tailored assistance. We compared its performance with alternative productivity-assistive tools, such as dashboard and narrative, in a study involving 40 participants. Our findings highlight the importance of user-centric design, adaptability, and the balance between personalization and privacy in AI-assisted productivity tools. By building on these insights, our work provides important guidance for developing more effective productivity solutions, ultimately leading to optimized efficiency and user experiences for information workers.

The future of work is rapidly changing, with remote and hybrid settings blurring the boundaries between professional and personal life. To understand how work rhythms vary across different work settings, we conducted a month-long study of 65 software developers, collecting anonymized computer activity data as well as daily ratings for perceived stress, productivity, and work setting. In addition to confirming the double-peak pattern of activity at 10:00 am and 2:00 pm observed in prior research, we observed a significant third peak around 9:00 pm. This third peak was associated with higher perceived productivity during remote days but increased stress during onsite and hybrid days, highlighting a nuanced interplay between work demands and work settings. Additionally, we found strong correlations between computer activity, productivity, and stress, including an inverted U-shaped relationship where productivity peaked at around six hours of computer activity before declining on more active days. These findings provide new insights into evolving work rhythms and highlight the impact of different work settings on productivity and stress.

We present AImagery, an AI-powered immersive relaxation experience tailored to user preferences and physiological feedback. In a study with 32 participants, half of them experienced a multisensory experience with scent, biofeedback, and a personalized AI audio story based on their heart rate, self-reported mood and custom scenery. The experimental group showed a significant anxiety reduction for those with moderate to high anxiety, as opposed to the control group (non-guided meditation/baseline resting control condition). User feedback was positive and received high ratings for enjoyment, immersion and, to a lesser extend, sleepiness. Our results highlight the potential of multisensory AI-driven relaxation tools for those with elevated anxiety.

Public speaking experts intentionally take their audience on an emotional roller coaster, staying attuned to their audience's collective emotional feedback. In this research, we explore how bidirectional sharing of heart rates between a speaker and their audience facilitates this emotional exchange, through empathy, emotional awareness, and engagement. Firstly, in two design studies$(N=25)$, we evaluated this concept and identified design elements for a heart rate sharing interface. Subsequently, we developed Feeling-the-Beat, a system for sharing heart rate between speakers and audiences in real time. Finally, in a randomized, counter-balanced, within-subjects study$(N=36)$, we compared our system to one sharing fabricated heart rates and a baseline system without heart rate sharing. Feeling-the-Beat significantly increased audience empathy towards the speaker, audience engagement during moments of heightened speaker heart rate, and social presence.

As concerns about employee burnout and skilled staff shortages in cybersecurity grow, our study aims to better understand the contributing factors to burnout in this field. Utilizing a mixed-methods approach, we analyze self-reported job and personal characteristics, along with digital activity data from 35 incident responders, identifying several factors such as high workload, time pressure, and lack of support from management. Our findings reveal that over half of the participants experience burnout (N=19), which is linked to increased workload, limited control, poor teamwork, and inadequate recognition. Burned-out responders often work more than 40 hours per week, have poor sleep quality, and engage in more email activities, meetings, and after-hour collaborations. Through our research, we also identify coping strategies individuals use to mitigate these stressors. Based on our findings, we provide practical recommendations to help organizations better support their cybersecurity incident response teams. While our study acknowledges limitations and suggests future research directions, it contributes significantly to understanding the challenges faced by cybersecurity incident responders. Our insights offer a comprehensive understanding of burnout factors in this domain and have broader implications for other high-stress work environments consistent with the interdisciplinary nature of CSCW.

Previous research suggests that biomusic, a type of biosignal sharing, is effective at promoting empathy and closeness among individuals. However, it is unclear whether these effects are due to the information it encodes or other emotional aspects of its resulting music. To explore this question, we developed a Generative Adversarial Network (GAN) to create synthetic biomusic that approximates real biomusic, and employed deception to evaluate its effects on 24 pairs of participants engaged in real-time emotional disclosure. Users reported that both real and synthetic biomusic provided the same amount of information about their conversational partner as observing body language, facial expressions, or vocal tone. Further, both conditions increased users’ ratings of closeness and empathy with each other compared to listening to no music. However, we found no statistically significant differences between the two biomusic conditions across any of our metrics. We discuss the implications of these results for the design of future biomusic systems.

Information workers often struggle to balance their time for a variety of activities like focused work, communication, and caring. This study analyzes the impact of a commercially available computer-assisted time protection intervention that automatically and preemptively schedules calendar time for self-determined activities. We analyzed the behaviors and self-reports of workers in two naturalistic studies. First, we studied 27 workers who were already using Computer-Assisted Protected Time (CAP time) and found that they mainly used it for focused work. Second, we analyzed the effect of CAP time as a randomized intervention on 89 workers who never had CAP time and found that those with it self-reported an increase in performance, job resources, and immersion. In both studies, workers with CAP time exhibited a rearrangement of activities leading to an overall reduction in work activity. This study highlights new opportunities for intelligent time-management interventions and the importance of protected time at work.

Deployment of facial action unit recognition models has been impeded due to their limited generalization to unseen people and demographics. This work conducts an in-depth generalization analysis across several sources of variance: individuals (40 subjects), genders (male and female), skin types (darker and lighter), and databases (BP4D and DISFA). To help suppress the variance in data, we propose using self-supervised denoising autoencoders to transfer facial expressions of different people onto a common facial template which is then used to train and evaluate each of the models. We show that person-independent models yielded significantly lower performance (55% average F1 and accuracy across 40 subjects) than person-dependent models (60.3 %), leading to a generalization gap of 5.3%. However, normalizing the data with the proposed method significantly increased the performance of person-independent models (59.6%). Similarly, the proposed method was able to significantly reduce the generalization gap when considering gender (2.4%), skin type (5.3%), and dataset (9.4%). These findings represent an important step towards the creation of more generalizable facial action unit recognition systems.

Workplace stress has been increasing in recent decades and has worsened by the unique demands imposed by COVID-19 and the new remote/hybrid work settings. High-stress working conditions can be detrimental to the health and wellness of workers and can lead to significant business costs in terms of productivity loss and medical expenses. An essential step toward managing stress involves finding comfortable ways to sense workers and recognizing stress as soon as it happens. This work explores the potential value of using pervasive sensors such as keyboards, webcams, and behavioral data such as calendar and e-mail activity to passively assess individual stress levels of work in real-life. In particular, we collected a large corpus of such data from 46 remote information workers over one month and asked them to self-report their stress levels and other relevant factors several times a day. Analysis of the data demonstrates that passive sensors can effectively detect both triggers and manifestations of workplace stress and that having access to prior data of the worker is critical for developing well-performing stress recognition models. Furthermore, we provide qualitative feedback capturing workers' preferences in workplace stress monitoring.

Workplace stress-reduction interventions have produced mixed results due to engagement and adherence barriers. Leveraging technology to integrate such interventions into the workday may address these barriers and help mitigate the mental, physical, and monetary effects of workplace stress. To inform the design of a workplace stress-reduction intervention system, we conducted a four-week longitudinal study with 86 participants, examining the effects of intervention type and timing on usage, stress reduction impact, and user preferences. We compared three intervention types and two delivery timing conditions: Pre-scheduled (PS) by users and Just-in-time (JIT) prompted by the system-identified user stress-levels. We found JIT participants completed significantly more interventions than PS participants, but post-intervention and study-long stress reduction was not significantly different between conditions. Participants rated low-effort interventions highest, but high-effort interventions reduced the most stress. Participants felt JIT provided accountability but desired partial agency over timing. We present type and timing implications.

The use of cameras and computational algorithms for noninvasive, low-cost and scalable measurement of physiological (e.g., cardiac and pulmonary) vital signs is very attractive. However, diverse data representing a range of environments, body motions, illumination conditions and physiological states is laborious, time consuming and expensive to obtain. Synthetic data have proven a valuable tool in several areas of machine learning, yet are not widely available for camera measurement of physiological states. Synthetic data offer "perfect" labels (e.g., without noise and with precise synchronization), labels that may not be possible to obtain otherwise (e.g., precise pixel level segmentation maps) and provide a high degree of control over variation and diversity in the dataset. We present SCAMPS, a dataset of synthetics containing 2,800 videos (1.68M frames) with aligned cardiac and respiratory signals and facial action intensities. The RGB frames are provided alongside segmentation maps and precise descriptive statistics about the underlying waveforms, including inter-beat interval, heart rate variability, and pulse arrival time. Finally, we present baseline results training on these synthetic data and testing on real-world datasets to illustrate generalizability.

Society has witnessed a rapid increase in the adoption of commercial uses of emotion recognition. Tools that were traditionally used by domain experts are now being used by individuals who are often unaware of the technology’s limitations and may use them in potentially harmful settings. The change in scale and agency, paired with gaps in regulation, urge the research community to rethink how we design, position, implement and ultimately deploy emotion recognition to anticipate and minimize potential risks. To help understand the current ecosystem of applied emotion recognition, this work provides an overview of some of the most frequent commercial applications and identifies some of the potential sources of harm. Informed by these, we then propose 12 guidelines for systematically assessing and reducing the risks presented by emotion recognition applications. These guidelines can help identify potential misuses and inform future deployments of emotion recognition.

The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N=175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N=117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.

Video-conferencing is essential for many companies, but its limitations in conveying social cues can lead to ineffective meetings. We present MeetingCoach, an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Through an exploratory survey (N=120), we identified important signals (e.g., turn taking, sentiment) and used these insights to create a wireframe dashboard. The design was evaluated with in situ participants (N=16) who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, we developed an AI system to quantify the meeting features and created personalized dashboards for each participant. Through interviews and surveys (N=23), we found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity. Based on our findings, we provide suggestions for future feedback system designs of video-conferencing meetings.

We present an automated framework for detecting Parkinson's disease (PD) from videos collected through a scalable online platform. We analyzed 1380 videos of age-matched participants performing four standard motor tasks from the MDS-UPDRS. Our proposed framework leverages multiple deep neural networks to temporally and spatially segment the videos as well as magnify relevant motions. Frequency domain representations of the resulting data are then classified using supervised learning. Overall, the proposed framework achieves an accuracy of 82.5% when discriminating between those with PD and those without, and 61.8% when discriminating between those with PD with treatment, with PD without treatment, and those without PD. These results increased up to 91.8% and 73.5%, respectively, when combining the predictions of multiple models. To understand the contributions of each part of our framework we perform systematic ablation studies. We also compare between motion features based on pixel, phase-based and deep learning-based representations. This work demonstrates the possibility of identifying PD cues in challenging real-life settings with inexpensive webcams.

Driving can occupy a considerable part of our daily lives and is often associated with high levels of stress. Motivated by the effectiveness of controlled breathing, this work studies the potential use of breathing interventions while driving to help manage stress. In particular, we implemented and evaluated a closed-loop system that monitored the breathing rate of drivers in real-time and delivered either a conscious or an unconscious personalized acoustic breathing guide whenever needed. In a study with 24 participants, we observed that conscious interventions more effectively reduced the breathing rate but also increased the number of driving mistakes. We observed that prior driving experience as well as personality are significantly associated with the effect of the interventions, which highlights the importance of considering user profiles for in-car stress management interventions.

Several research projects have recently explored the use of physiological sensors such as electroencephalography (EEG) or electrooculography (EOG) to measure the engagement and vigilance of a user in context of car driving. However, these systems still suffer from limitations such as an absence of a socially acceptable form-factor and use of impractical, gel-based electrodes. We present AttentivU, a device using both EEG and EOG for real-time monitoring of physiological data. The device is designed as a socially acceptable pair of glasses and employs silver electrodes. It also supports real-time delivery of feedback in the form of an auditory signal via a bone conduction speaker embedded in the glasses. A detailed description of the hardware design and proof of concept prototype is provided, as well as preliminary data collected from 20 users performing a driving task in a simulator in order to evaluate the signal quality of the physiological data.

This paper studies the feasibility of using low-cost motion sensors to provide opportunistic heart rate assessments from ballistocardiographic signals during restful periods of daily life. Three wearable devices were used to capture peripheral motions at specific body locations (head, wrist, and trouser pocket) of 15 participants during five regular workdays each. Three methods were implemented to extract heart rate from motion data and their performance was compared to those obtained with an FDA-cleared device. With a total of 1358 h of naturalistic sensor data, our results show that providing accurate heart rate estimations from peripheral motion signals is possible during relatively "still" moments. In our real-life workplace study, the head-mounted device yielded the most frequent assessments (22.98% of the time under 5 beats per minute of error) followed by the smartphone in the pocket (5.02%) and the wrist-worn device (3.48%). Most importantly, accurate assessments were automatically detected by using a custom threshold based on the device jerk. Due to the pervasiveness and low cost of wearable motion sensors, this paper demonstrates the feasibility of providing opportunistic large-scale low-cost samples of resting heart rate.

Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images.

A growing number of studies show links between changes in tongue appearance and human health conditions. This paper studies tongue color changes in the context of stress to explore the feasibility of providing a novel and non-invasive stress measurement method. In a laboratory study, 24 participants were asked to perform a calm and a stressful math task and to take a photo of their tongue right after each of the tasks. We observed subtle but consistent color differences between calm and stress tasks for up to 75% of the participants, which was consistent with both self-report and physiological metrics of stress. Moreover, we observed significant correlations of up to 0.72 between certain tongue colors and long-term stress assessed with the 10-item Perceived Stress Scale questionnaire. We discuss the potential implications of this work and highlight some lines of future research.

Contact-free camera-based measurement of cognitive stress opens up new possibilities for human-computer interaction with applications in remote learning, stress monitoring, and optimization of workload for user experience. The autonomic nervous system controls the inter-beat intervals of the heart and breathing patterns, and these signals change under cognitive stress. We built a participant-independent cognitive stress recognition model based on photoplethysmographic signals measured remotely at a distance of 3 meters. We tested the model on naturalistic responses from 10 individuals completing randomized-order computer-based tasks (ball control and card sorting). The system successfully detected increased stress during the tasks, which were consistent with self-report measures. Changes in heart rate variability were more discriminative indicators of cognitive stress than were heart rate and breathing rate.

The Experience Sampling Method is widely used for collecting self-report responses from people in natural settings. While most traditional approaches rely on using a phone to trigger prompts and record information, wearable devices now offer new opportunities that may improve this method. This research quantitatively and qualitatively studies the experience sampling process on head-worn and wrist-worn wearable devices, and compares them to the traditional "smartphone in the pocket." To enable this work, we designed and implemented a custom application to provide similar prompts across the three types of devices and evaluated it with 15 individuals for five days (75 days total), in the context of real-life stress measurement. We found significant differences in response times across devices, and captured tradeoffs in interaction types, screen size, and device familiarity that can affect both users' experience and the reports made by users.

During recent years a large variety of wearable devices have become commercially available. As these devices are in close contact with the body, they have the potential to capture sensitive and unexpected personal data even when the wearer is not moving. This work demonstrates that wearable motion sensors such as accelerometers and gyroscopes embedded in head-mounted and wrist-worn wearable devices can be used to identify the wearer (among 12 participants) and his/her body posture (among 3 positions) from only 10 seconds of “still” motion data. Instead of focusing on large and apparent motions such as steps or gait, the proposed methods amplify and analyze very subtle body motions associated with the beating of the heart. Our findings have the potential to increase the value of pervasive wearable motion sensors but also raise important privacy concerns that need to be considered.

Recognizing when computer users are stressed can help reduce their frustration and prevent a large variety of negative health conditions associated with chronic stress. However, measuring stress non-invasively and continuously at work remains an open challenge. This work explores the possibility of using a pressure-sensitive keyboard and a capacitive mouse to discriminate between stressful and relaxed conditions in a laboratory study. During a 30 minute session, 24 participants performed several computerized tasks consisting of expressive writing, text transcription, and mouse clicking. During the stressful conditions, the large majority of the participants showed significantly increased typing pressure (>79% of the participants) and more contact with the surface of the mouse (75% of the participants). We discuss the potential implications of this work and provide recommendations for future work.

The recent emergence of comfortable wearable sensors has focused almost entirely on monitoring physical activity, ignoring opportunities to monitor more subtle phenomena, such as the quality of social interactions. We argue that it is compelling to address whether physiological sensors can shed light on quality of social interactive behavior. This work leverages the use of a wearable electrodermal activity (EDA) sensor to recognize ease of engagement of children during a social interaction with an adult. In particular, we monitored 51 child-adult dyads in a semi-structured play interaction and used Support Vector Machines to automatically identify children who had been rated by the adult as more or less difficult to engage. We report on the classification value of several features extracted from the child's EDA responses, as well as several other features capturing the physiological synchrony between the child and the adult.

In this study, we created and evaluated a computer vision based system that automatically encouraged, recognized and counted smiles on a college campus. During a ten-week installation, passersby were able to interact with the system at four public locations. The aggregated data was displayed in real time in various intuitive and interactive formats on a public website. We found privacy to be one of the main design constraints, and transparency to be the best strategy to gain participants' acceptance. In a survey (with 300 responses), participants reported that the system made them smile more than they expected, and it made them and others around them feel momentarily better. Quantitative analysis of the interactions revealed periodic patterns (e.g., more smiles during the weekends) and strong correlation with campus events (e.g., fewer smiles during exams, most smiles the day after graduation), reflecting the emotional responses of a large community.

Individuals diagnosed with Autism Spectrum Disorders (ASD) often have challenging behaviors (CB's), such as self-injury or emotional outbursts, which can negatively impact the quality of life of themselves and those around them. Recent advances in mobile and ubiquitous technologies provide an opportunity to efficiently and accurately capture important information preceding and associated with these CB's. The ability to obtain this type of data will help with both intervention and behavioral phenotyping efforts. Through collaboration with behavioral scientists and therapists, we identified relevant design requirements and created an easy-to-use mobile application for collecting, labeling, and sharing in-situ behavior data in individuals diagnosed with ASD. Furthermore, we have released the application to the community as an open-source project so it can be validated and extended by other researchers.