This demo presents CO-PARLEY, a mobile socially interactive agent designed to support individuals experiencing difficulties with emotion regulation.The system engages users through reciprocal coregulation, a dynamic and two-way process in which the agent and user mutually influence each other's emotional and physiological states.By combining verbal and nonverbal interaction with real-time physiological synchrony, CO-PARLEY fosters therapeutic alliance, trust, and emotional awareness.Built on a modular framework that integrates multimodal sensing, dialogue management, and adaptive behavior generation, the agent supports users in emotionally challenging moments.In general, this alliance can improve the effectiveness of psychoeducational and awareness exercises.

This paper introduces SIA-Lab, a versatile and broadly applicable platform to advance socially interactive agents (SIAs).Unlike existing single-use case systems, SIA-Lab is a modular and scalable framework built on Android and mobile technologies, enabling rapid explorative and comparative studies between human-human and human-agent interactions, offering valuable insights into behavioral dynamics and user engagement.Effective in health-related applications, including screening, therapeutic assistance, and posttreatment care, it is equally adaptable to education and other settings.By integrating dialog management, affective modeling, and multimodal interaction analysis, SIA-Lab represents a comprehensive toolkit for researchers and practitioners to evaluate and design next-generation supportive technologies.

Computational emotion recognition relies on observable expressions. However, negative situations can evoke regulation mechanisms that obscure and mask emotional experiences, often by smiling. As smiles are typically associated with positive emotions, this mismatch of emotional experience and expression may lead to misinterpretations by most current algorithmic affective computing approaches. To improve computational modeling of real-life experiences and expressions in negative social situations, we explore connections between smile appearance and function, incorporating participants' rich personal self-reports into ground truth labels for their expressions. We present an empirically grounded smile corpus of 199 smiles that is based on a) recordings of N = 30 participants in negative social situations that are analyzed regarding smile morphology and b) a category system of smile functions based on participants‘ self-reports. In a computational model, we used cleaned corpus data of 183 unique smile instances to classify five smile function categories based on observable nonverbal signals, with results benchmarked at above chance. Applying a theory- and data-driven approach, our analyses confirm a complex relationship between internal smile functions and observable signals. Finally, we discuss smile functions in negative social situations, including ‘despising’, ‘provoking’, and 'kiss my ass'-smiles.

Human emotions are often not expressed directly, but regulated according to internal processes and social display rules. For affective computing systems, an understanding of how users regulate their emotions can be highly useful, for example to provide feedback in job interview training, or in psychotherapeutic scenarios. However, at present no method to automatically classify different emotion regulation strategies in a cross-user scenario exists. At the same time, recent studies showed that instruction-tuned Large Language Models (LLMs) can reach impressive performance across a variety of affect recognition tasks such as categorical emotion recognition or sentiment analysis. While these results are promising, it remains unclear to what extent the representational power of LLMs can be utilized in the more subtle task of classifying users' internal emotion regulation strategy. To close this gap, we make use of the recently introduced Deep corpus for modeling the social display of the emotion shame, where each point in time is annotated with one of seven different emotion regulation classes. We fine-tune Llama2-7B as well as the recently introduced Gemma model using Low-rank Optimization on prompts generated from different sources of information on the Deep corpus. These include verbal and nonverbal behavior, person factors, as well as the results of an indepth interview after the interaction. Our results show, that a fine-tuned Llama2-7B LLM is able to classify the utilized emotion regulation strategy with high accuracy (0.84) without needing access to data from post-interaction interviews. This represents a significant improvement over previous approaches based on Bayesian Networks and highlights the importance of modeling verbal behavior in emotion regulation.

Understanding emotions is key to Affective Computing. Emotion recognition focuses on the communicative component of emotions encoded in social signals. This view alone is insufficient for a deeper understanding and computational representation of the internal, subjectively experienced component of emotions. This paper presents a cognition-based method calledDeepas a starting point for deeper computational modeling of the internal component of emotions.Deepincorporates an approach to query individual internal emotional experiences and to represent such information computationally. It combines social signals, verbalized introspection information, context information, and theory-driven knowledge. We apply theDeepmethod to the emotion of shame as an example and compare it to a typical emotion recognition model, highlighting the differences and advantages.

This study evaluates a socially interactive agent to create an embodied cobot. It tests a real-time continuous emotional modeling method and an aligned transparent behavioral model, BASSF (boredom, anxiety, self-efficacy, self-compassion, flow). The BASSF model anticipates and counteracts counterproductive emotional experiences of operators working under stress with cobots on tedious tasks. The flow experience is represented in the three-dimensional pleasure, arousal, and dominance (PAD) space. The embodied covatar (cobot and avatar) is introduced to support flow experiences through emotion regulation guidance. The study tests the model's main theoretical assumptions about flow, dominance, self-efficacy, and boredom. Twenty participants worked on a task for an hour, assembling pieces in collaboration with the covatar. After the task, participants completed questionnaires on flow, their affective experience, and self-efficacy, and they were interviewed to understand their emotions and regulation during the task. The results suggest that the dominance dimension plays a vital role in task-related settings as it predicts the participants' self-efficacy and flow. However, the relationship between flow, pleasure, and arousal requires further investigation. Qualitative interview analysis revealed that participants regulated negative emotions, like boredom, also without support, but some strategies could negatively impact well-being and productivity, which aligns with theory.

Humans can develop closeness through the exchange of personal information. A structured method of self-disclosure has been developed in the Fast Friends paradigm, in which two people alternately ask 36 questions with increasing levels of interpersonal intimacy. We transferred this paradigm to interactions with Socially Interactive Agents (SIA). In our study, 72 participants alternately asked and answered 36 questions with a SIA -- indicating their level of interpersonal closeness with the SIA at three points. Participants rated specific trust in the SIA after the interaction, and their general trust and attachment styles were measured. Over time, participants' levels of closeness increased, which was moderated by specific trust but not by general trust and attachment style. Participants with high specific trust developed higher levels of closeness to the SIA than participants with low specific trust. These findings indicate that people can develop a close relationship with SIAs and that trust in the SIA is a prerequisite for developing closeness. Furthermore, this paper introduced the Inclusion of Other in the Self Scale for assessing the relationship between two interaction partners during an ongoing interaction.

Understanding emotions is key to Affective Computing. Emotion recognition focuses on the communicative component of emotions encoded in social signals. This view alone is insufficient for deeper understanding and computational representation of the internal, subjectively experienced component of emotions. This paper presents the Deep method as a starting point for a deeper computational modeling of internal emotions. The method includes how to query individual internal emotional experiences, and it shows an approach to represent such information computationally. It combines social signals, verbalized introspection information, context information, and theory-driven knowledge. We apply the Deep method exemplary on the emotion shame and present a schematic dynamic Bayesian network for modeling it.

In the last decade, empirical sciences have faced a tremendous change in the way of conducting research. As a broad interdisciplinary field, research in Affective Computing often employs empirical user studies. The current paper analyzes research practices in Affective Computing and deduces recommendations for improving the quality of methods and reporting. We extracted a total of k = 65 empirical studies from the two most recent International Conferences on Affective Computing & Intelligent Interaction (ACII) ’17 and ’19. Three raters summarized characteristics of studies (e.g., number of experimental studies) and how much methodological (e.g., participant characteristics) and statistical information (e.g., degrees of freedom) were missing. Also, we conducted a p-curve analysis to test the overall evidential value of findings. Results showed that 1. in at least half of the studies, one important information about statistical results was missing, and 2. those k = 31 studies that had reported all necessary information to be included into the p-curve showed evidential value. In general, all criteria were never met in one single study. We provide concrete recommendations on how to implement open research practices for empirical studies in Affective Computing.

Coping with stress is critical to mental health. Prolonged mental stress is the psychological and physiological response to a high frequency of or continuous stressors, which has a negative impact on health. This paper presents a virtual stress management training using biofeedback derived from the cardiovascular response of the heart rate variability (HRV) with an interactive social agent as biofeedback trainer. The evaluation includes both, a subject-matter expert interview and an experiment with 71 participants. In the experiment, we compared our novel stress management training to a stress management training using stress diaries. The results indicate that our social agent-based stress management training using biofeedback significantly decreased the self-assessed stress levels immediately after the training, as well as in a socially stressful task. Moreover, we found a significant correlation between stress level and the assessment of one’s performance in a socially stressful task. Participants that received our training assessed their performance higher than participants getting stress diaries. Taken this together, our novel virtual stress management training with an interactive social agent as a trainer can be evaluated as a valid method for learning techniques on how to cope with stressful situations.

Virtual agents play an important role when we interact with machines. They are in the role of assistants or companions with less or more human-like appearance. Such agents influence our behavior. With an increasing and broader distribution, their influence might become stronger, and at some point, they might even adopt roles with a degree of authority. This paper presents the results of a study that examines the obedience of human users towards a) an embodied virtual agent in the role of an instructor and b) a human in the role of an instructor. Under a cover-story of a creativity test, participants should fulfill stressful and shameful tasks. Our results indicate that the embodied virtual agent has the same authority as the human instructor. The agent is also able to elicit the same level of the negative feelings stress and shame.

This paper presents a study that examines whether social agents can elicit the social emotion shame as humans do. For that, we use job interviews, which are highly evaluative situations per se. We vary the interview style (shame-eliciting vs. neutral) and the job interviewer (human vs. social agent). Our dependent variables include observational data regarding the social signals of shame and shame regulation as well as self-assessment questionnaires regarding the felt uneasiness and discomfort in the situation. Our results indicate that social agents can elicit shame to the same amount as humans. This gives insights about the impact of social agents on users and the emotional connection between them.

Using Social Agents as health-care assistants or trainers is one focus area of IVA research. This paper presents a concept of our mobile Social Agent EmmA in the role of a vocational reintegration assistant for burn-out outpatient treatment. We follow a typical par- ticipatory design approach including experts and patients in order to address requirements from both sides. Since the success of such treatments is related to a patients emotion regulation capabilities, we employ a real-time social signal interpretation together with a computational simulation of emotion regulation that influences the agent's social behavior as well as the situational selection of verbal treatment strategies. Overall, our interdisciplinary approach sketches a novel integrative concept for Social Agents as assistants for burn-out patients.

Human interaction partners can deal with interruptions and then resume the interaction. This ability should be emulated by social agents. How fast interruptions are handled might influence the overall impression of an agent. In this paper, we present the results of a user study on how a human dialog partner perceives the be- havior of a virtual agent handling verbal user interruptions with different reaction times. The study goes beyond typical perception experiments by preserving the real-time interaction experience. For the evaluation, we rely on a parametrizable parallelized computa- tional model that represents dialog flow, overlap detection, conflict recognition, and conflict handling in real-time. The evaluation re- sults show that the timing of the agent's interruption handling in interactive human-agent dialogues is related to different interper- sonal attitudes.

Virtual agents usually come in a virtual environment that can be designed in various ways which might affect users. This paper presents a study that examines whether the design of the virtual environment has an impact on the assessment of the virtual agent and the interaction. In a virtual job interview training, participants interacted with a virtual interviewer that behaved exactly the same, but the background and lighting conditions were manipulated. Our results indicate that the environmental design affects the assessment of the interviewer as well as the interview process.

In this paper, we focus on experience-based role play with virtual agents to provide young adults at the risk of exclusion with social skill training. We present a scenario-based serious game simulation platform. It comes with a social signal interpretation component, a scripted and autonomous agent dialog and social interaction behavior model, and an engine for 3-D rendering of lifelike virtual social agents in a virtual environment. We show how two training systems developed on the basis of this simulation platform can be used to educate people in showing appropriate socioemotive reactions in job interviews. Furthermore, we give an overview of four conducted studies investigating the effect of the agents' portrayed personality and the appearance of the environment on the players' perception of the characters and the learning experience.

This paper presents an automotive interaction concept called GetHomeSafe that allows drivers to book hotels, browse news and use both Wolfram Alpha and Facebook safely during driving. GetHomeSafe includes a natural speech dialog system and a graphical user interface both adapted to drivers' needs. In addition to a description of the interaction concept, we report the results of a real-car driving study. To assess the driving safety subjective and objective data on GetHomeSafe usage were gathered and compared with the usage of respective apps on a mounted tablet. In order to combine information about eye-gaze and head-pose of the driver, we used the EyeVIUS system [1] to get the drivers' focus-of-attention. The results indicate that it seems to be safer to use GetHomeSafe compared to the apps on a mounted tablet. One reason is that the subjective load and the distraction while using the mounted tablet is higher for most of the drivers. Moreover, the GetHomeSafe interaction concept received favorable usability ratings.

Intelligent technologies have been used in various ways to support more effective representation and processing of media and documents in terms of the events that they refer to. This demo presents some innovations that have been introduced in a web-based interface to a repository of media and documents that are organized in terms of hierarchically structured events.

In this paper, we describe a technique to determine user preferences concerning in-car micro-gesture interaction. The approach is derived from the theater technique [1], and implies a collaborative adjustment of parameters with the experimenter, until the subject has decided about the final settings. We evaluated three systematically selected gestures (zooming, sweeping, and circling) for controlling four exemplary comfort functions of the car (window lifter, air condition, radio volume, and seat heating). The main result of our study is the geometry of a "sweet spot" for micro-gesture recognition close to the steering wheel, which is independent from the underlying technical recognition approach. Additionally, preferred sizes, angles, and pause times for the investigated gestures are provided. We give an indication, which of the gestures is preferred by the users (the sweeping gesture). Finally, we provide a more detailed view on the interaction between gesture preferences and function.