Visual representations of "the user" are a key part of academic papers in human-robot interaction (HRI). These visualizations can include photos of participants, drawings of users, or simulated personas. Critical analyses have revealed representation biases in scholarly work, often detectable in formal output like publications. Sampling biases, limitations in demographic reporting, and bias across researchers have been discovered. Yet, no work to date has considered the visualizations (drawings, photos, and other graphical representations) of the human side in the HRI equation. We surface representation biases in work from the flagship ACM/IEEE HRI conference: over-representation of younger light-skinned men with typical bodies, and under-representation of people with darker skin tones, women and gender-diverse people, people with disabilities, and people of various ages and sizes. We critically discuss these trends and offer suggestions for best practices in reporting.

The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing “narrative captions” relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on “Who”, “What”, “Where” and “How” questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLa Va, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the “Fast” narrative descriptors and “Slow” reasoning of language models is a promising way to achieve emotional theory of mind.

How does the person in the bounding box feel?" Achieving human-level recognition of the apparent emotion of a person in real world situations remains an unsolved task in computer vision. Facial expressions are not enough: body pose, contextual knowledge, and commonsense reasoning all contribute to how humans perform this emotional theory of mind task. In this paper, we examine two major approaches enabled by recent large vision language models: 1) image captioning followed by a language-only LLM, and 2) vision language models, under zero-shot and fine-tuned setups. We evaluate the methods on the Emotions in Context (EMOTIC) dataset and demonstrate that a vision language model, fine-tuned even on a small dataset, can significantly outperform traditional baselines. The results of this work aim to help robots and agents perform emotionally sensitive decision-making and interaction in the future.

Dialogue systems are increasingly recognized as valuable assets in healthcare, aiding in the support of individuals grappling with mental health challenges. In this paper, we develop and evaluate a task-oriented empathic chatbot, EmoBot, in a mental healthcare patient intake scenario. EmoBot employs Retrieval-Augmented Generation (RAG) methods to provide guardrails to Large Language Models (LLMs) in providing targeted empathic support, while ensuring task-completion of administering PHQ-9 intake questionnaire. We evaluate EmoBot’s performance by using a simulated patient model to mimic varying levels of depression severity. Our preliminary results showed high task completion rates and agreement between EmoBot’s depression severity categorization and human raters, demonstrating the promise of RAG-based methods in guiding LLMs to be used for health assessment and support.

The IDC research community has a growing interest in designing AI interfaces for children with special educational needs. Nonetheless, little research has explored the research and design issues, rationale, challenges, and opportunities in this field. Therefore, we propose to host a half-day workshop to bring together researchers and practitioners from the Learning & Education, Accessibility, and Intelligent User Interfaces sub-fields to discuss and identify existing design issues, challenges, and collaboration barriers, to establish consensus on the design of a pragmatic framework, as well as explore future innovation and research opportunities. We aim to foster mutual understanding and in-depth collaboration among researchers in the IDC community.

Abstract Recent advances in reinforcement learning with social agents have allowed such models to achieve human-level performance on certain interaction tasks. However, most interactive scenarios do not have performance alone as an end-goal; instead, the social impact of these agents when interacting with humans is as important and largely unexplored. In this regard, this work proposes a novel reinforcement learning mechanism based on the social impact of rivalry behavior. Our proposed model aggregates objective and social perception mechanisms to derive a rivalry score that is used to modulate the learning of artificial agents. To investigate our proposed model, we design an interactive game scenario, using the Chef’s Hat Card Game, and examine how the rivalry modulation changes the agent’s playing style, and how this impacts the experience of human players on the game. Our results show that humans can detect specific social characteristics when playing against rival agents when compared to common agents, which affects directly the performance of the human players in subsequent games. We conclude our work by discussing how the different social and objective features that compose the artificial rivalry score contribute to our results.

Research has shown that free-form Game-Design (GD) environments can be very effective in fostering Computational Thinking (CT) skills at a young age. However, some students can still need some guidance during the learning process due to the highly open-ended nature of these environments. Intelligent Pedagogical Agents (IPAs) can be used to provide personalized assistance in real-time to alleviate this challenge. This paper presents our results in evaluating such an agent deployed in a real-word free-form GD learning environment to foster CT in the early K-12 education, Unity-CT. We focus on the effect of repetition by comparing student behaviors between no intervention, 1-shot, and repeated intervention groups for two different errors that are known to be challenging in the online lessons of Unity-CT. Our findings showed that the agent was perceived very positively by the students and the repeated intervention showed promising results in terms of helping students make fewer errors and more correct behaviors, albeit only for one of the two target errors. Building from these results, we provide insights on how to provide IPA interventions in free-form GD environments.

Free-form Game-Design (GD) environments show promise in fostering Computational Thinking (CT) skills at a young age. However, such environments can be challenging to some students due to their highly open-ended nature. Our long-term goal is to alleviate this difficulty via pedagogical agents that can monitor the student interaction with the environment, detect when the student needs help and provide personalized support accordingly. In this paper, we present a preliminary evaluation of one such agent deployed in a real-word free-form GD learning environment to foster CT in the early K-12 education, Unity-CT. We focus on the effect of repetition by comparing student behaviors between no intervention, 1-shot, and repeated intervention groups for two different errors that are known to be challenging in the online lessons of Unity-CT environment. Our findings showed that the agent was perceived very positively by the students and the repeated intervention showed promising results in terms of helping students make less errors and more correct behaviors, albeit only for one of the two target errors. Based on these results, we provide insights on how to improve the delivery of the agent’s interventions in free-form GD environments.

Game-Design (GD) environments show promise in fostering Computational Thinking (CT) skills at a young age. However, such environments can be challenging to some students due to their highly open-ended nature. We propose to alleviate this difficulty by learning interpretable student models from data that can drive personalization of a real-world GD learning environment to the student’s needs. We apply our approach on a dataset collected in ecological settings and evaluate the ability of the generated student models at predicting ineffective learning behaviors over the course of the interaction. We then discuss how these behaviors can be used to define personalized support in GD learning activities, by conducting extensive interviews with experienced instructors.

Visual, behavioral and verbal cues for gender are often used in designing virtual agents to take advantage of their stereotypical effects on the users. However, recent studies point towards a more gender-balanced view of stereotypical traits and roles in our society. This paper is intended as an effort towards a progressive and inclusive approach for gender representations in virtual agents. Our contributions are two-fold. First, in an iterative design process, we created representative male, female and androgynous agents with few differences in their visual attributes. Second, we used these agents to evaluate the stereotypical assumptions of gendered traits and roles in virtual agents. Our results showed that, indeed, gender stereotypes are not as effective as previously assumed, and androgynous agents could represent a middle-ground between gendered stereotypes. We present our findings in the hope to foster discussions in virtual agent research and the frequent stereotypical use of gender representations.

Social Prescribing is a novel holistic approach in health care that focuses on the importance of socio-psychological factors in well-being to support physical well-being across the lifespan. Despite the promising results shown in several studies, there are many barriers in providing this solution to the masses, such as lack of wide-range availability of personalized solutions, cost and logistic issues. In this work, we propose using assistive technologies, namely social virtual agents for art prescription, to provide an easy-to-use, cost-effective and personalizable solution for older adults. We describe the details of our Empathic AI-Painter System and lay out the details of our design and evaluation methodology.

The novel research area of computational empathy is in its infancy and moving towards developing methods and standards. One major problem is the lack of agreement on the evaluation of empathy in artificial interactive systems. Even though the existence of well-established methods from psychology, psychiatry and neuroscience, the translation between these methods and computational empathy is not straightforward. It requires a collective effort to develop metrics that are more suitable for interactive artificial agents. This paper is aimed as an attempt to initiate the dialogue on this important problem. We examine the evaluation methods for empathy in humans and provide suggestions for the development of better metrics to evaluate empathy in artificial agents. We acknowledge the difficulty of arriving at a single solution in a vast variety of interactive systems and propose a set of systematic approaches that can be used with a variety of applications and systems.

This paper is intended to outline the PhD research that is aimed to model empathy in embodied conversational systems. Our goal is to determine the requirements for implementation of an empathic interactive agent and develop evaluation methods that is aligned with the empathy research from various fields. The thesis is composed of three scientific contributions: (i) developing a computational model of empathy, (ii) implementation of the model in embodied conversational agents and (iii) enhance the understanding of empathy in interaction by generating data and build evaluation tools. The paper will give results for the contribution (i) and preliminary results for contribution (ii). Moreover, we will present the future plan for contribution (ii) and (iii).