Recent advances in artificial intelligence (AI), including generative models and socially interactive agents, are reshaping the design of interactive systems and raising new challenges for Human–Computer Interaction (HCI). While AI research has traditionally focused on improving performance and scalability, HCI emphasizes usability, transparency, and the broader social implications of technology, all aspects that imply the application of proper human-centered design methods. Bridging these perspectives is increasingly critical as AI moves from backend functionality to a central role in user interaction. This paper reports on the workshop Future Horizons in Human–AI Interaction: Joint Perspectives from HCI and AI, held at AVI 2026. It synthesizes key insights on emerging challenges and opportunities in designing human-centered AI systems, emphasizing the importance of integrating perspectives from HCI and AI, to advance the design of interactive AI systems that are both technically robust and aligned with human values.

Recent research in affective robots has recognized their potential in supporting human well-being. Due to rapidly developing affective and artificial intelligence technologies, this field of research has undergone explosive expansion and advancement in recent years. In order to develop a deeper understanding of recent advancements, we present a systematic review of the past 10 years of research in affective robotics for wellbeing. In this review, we identify the domains of well-being that have been studied, the methods used to investigate affective robots for well-being, and how these have evolved over time. We also examine the evolution of the multifaceted research topic from three lenses: technical, design, and ethical. Finally, we discuss future opportunities for research based on the gaps we have identified in our review – proposing pathways to take affective robotics from the past and present to the future. The results of our review are of interest to human-robot interaction and affective computing researchers, as well as clinicians and well-being professionals who may wish to examine and incorporate affective robotics in their practices.

Recent advancements in robots powered by large language models have enhanced their conversational abilities, enabling interactions closely resembling human dialogue. However, these models introduce safety and security concerns in HRI, as they are vulnerable to manipulation that can bypass built-in safety measures. Imagining a social robot deployed in a home, this work aims to understand how everyday users try to exploit a language model to violate ethical principles, such as by prompting the robot to act like a life partner. We conducted a pilot study involving 21 university students who interacted with a Misty robot, attempting to circumvent its safety mechanisms across three scenarios based on specific HRI ethical principles: attachment, freedom, and empathy. Our results reveal that participants employed five techniques, including insulting and appealing to pity using emotional language. We hope this work can inform future research in designing strong safeguards to ensure ethical and secure human-robot interactions.

Group interactions are a natural part of our daily life, and as robots become more integrated into society, they must be able to socially interact with multiple people at the same time. However, group human-robot interaction (HRI) poses unique computational challenges often overlooked in the current HRI literature. We conducted a scoping review including 44 group HRI papers from the last decade (2015–2024). From these papers, we extracted variables related to perception and behaviour generation challenges, as well as factors related to the environment, group, and robot capabilities that influence these challenges. Our findings show that key computational challenges in perception included detection of groups, engagement, and conversation information, while challenges in behaviour generation involved developing approaching and conversational behaviours. We also identified research gaps, such as improving detection of subgroups and interpersonal relationships, and recommended future work in group HRI to help researchers address these computational challenges.

The integration of large language models (LLMs) into conversational robots has made human-robot conversations more dynamic. Yet, LLM-powered conversational robots remain prone to errors, e.g., misunderstanding user intent, prematurely interrupting users, or failing to respond altogether. Detecting and addressing these failures is critical for preventing conversational breakdowns, avoiding task disruptions, and sustaining user trust. To tackle this problem, the ERR@HRI 2.0 Challenge provides a multimodal dataset of LLM-powered conversational robot failures during human-robot conversations and encourages researchers to benchmark machine learning models designed to detect robot failures. The dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features. Each interaction is annotated with the presence or absence of robot errors from the system perspective, and perceived user intention to correct for a mismatch between robot behavior and user expectation. Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data. Submissions will be evaluated using various performance metrics, including detection accuracy and false positive rate. This challenge represents another key step toward improving failure detection in human-robot interaction through social signal analysis.

In dyadic interactions, a broad spectrum of human facial reactions might be appropriate for responding to each human speaker behaviour. Following the successful organisation of the REACT 2023 and REACT 2024 challenges, we are proposing the REACT 2025 challenge encouraging the development and benchmarking of Machine Learning (ML) models that can be used to generate multiple appropriate, diverse, realistic and synchronised human-style facial reactions expressed by human listeners in response to an input stimulus (i.e., audio-visual behaviours expressed by their corresponding speakers). As a key of the challenge, we provide challenge participants with the first natural and large-scale multi-modal Multiple Appropriate Facial Reaction Generation (MAFRG) dataset (called MARS) recording 136 human-human dyadic interactions containing a total of 2856 interaction sessions covering five different topics. In addition, this paper also presents the challenge guidelines and the performance of our baselines on the two proposed sub-challenges: Offline MAFRG and Online MAFRG, respectively. The challenge baseline code is publicly available at https://github.com/reactmultimodalchallenge/baseline_react2025

When children are anxious or scared, it can be hard for them to stay still or follow instructions during medical procedures, making the process more challenging and affecting procedure results. This is particularly true for radiological procedures, where long scan times, confined spaces, and loud noises can cause children to move, significantly impacting scan quality. To this end, sometimes children are sedated, but doctors are constantly seeking alternative non-pharmacological solutions. This work aims to explore how social robots could assist in preparing children for radiological procedures. We have conducted a focus group discussion with five hospital stakeholders, namely radiographers, pediatricians, and clinical engineers, to explore (i) the context regarding children’s preparation for radiological procedures, hence their needs and how children are currently prepared, and (ii) the potential role of social robots in this process. The discussion was transcribed and analysed using thematic analysis. Among our findings, we identified three potential roles for a social robot in this preparation process: offering infotainment in the waiting room, acting as a guide within the hospital, and assisting radiographers in preparing children for the procedure. We hope that insights from this study will inform the design of social robots for pediatric healthcare.

One of the primary goals of Human-Robot Interaction (HRI) research is to develop robots that can interpret human behavior and adapt their responses accordingly. Adaptive learning models, such as continual and reinforcement learning, play a crucial role in improving robots’ ability to interact effectively in real-world settings. However, these models face significant challenges due to the limited availability of real-world data, particularly in sensitive domains like healthcare and well-being. To address these challenges, causality provides a structured framework for understanding and modeling the underlying relationships between actions, events, and outcomes. By moving beyond mere pattern recognition, causality enables robots to make more explainable and generalizable decisions. This paper presents an exploratory causality-based analysis through a case study of an adaptive robotic coach delivering positive psychology exercises over four weeks in a workplace setting. The robotic coach autonomously adapts to multimodal human behaviors, such as facial valence and speech duration. By conducting both macro- and micro-level causal analyses, this study aims to gain deeper insights into how adaptability can enhance well-being during interactions. Ultimately, this research seeks to advance our understanding of how causality can help overcome challenges in HRI, particularly in real-world applications.

Recently, several works have explored if and how robotic coaches can promote and maintain mental well-being in different settings. However, findings from these studies revealed that these robotic coaches are not ready to be used and deployed in real-world settings due to several limitations that span from technological challenges to coaching success. To overcome these challenges, this article presents VITA, a novel multi-modal LLM-based system that allows robotic coaches to autonomously adapt to the coachee’s multi-modal behaviours (facial valence and speech duration) and deliver coaching exercises in order to promote mental well-being in adults. We identified five objectives that correspond to the challenges in the recent literature, and we show how the VITA system addresses these via experimental validations that include one in-lab pilot study ( N = 4) that enabled us to test different robotic coach configurations (pre-scripted, generic and adaptive models) and inform its design for using it in the real world, and one real-world study ( N = 17) conducted in a workplace over 4 weeks. Our results show that: (i) coachees perceived the VITA adaptive and generic configurations more positively than the pre-scripted one, and they felt understood and heard by the adaptive robotic coach, (ii) the VITA adaptive robotic coach kept learning successfully by personalising to each coachee over time and did not detect any interaction ruptures during the coaching and (iii) coachees had significant mental well-being improvements via the VITA-based robotic coach practice. The code for the VITA system is openly available via https://github.com/Cambridge-AFAR/VITA-system .

In dyadic interaction, predicting the listener's facial reactions is challenging as different reactions could be appropriate in response to the same speaker's behaviour. Previous approaches predominantly treated this task as an interpolation or fitting problem, emphasizing deterministic outcomes but ignoring the diversity and uncertainty of human facial reactions. Furthermore, these methods often failed to model short-range and long-range dependencies within the interaction context, leading to issues in the synchrony and appropriateness of the generated facial reactions. To address these limitations, this paper reformulates the task as an extrapolation or prediction problem, and proposes an novel framework (called ReactFace) to generate multiple different but appropriate facial reactions from a speaker behaviour rather than merely replicating the corresponding listener facial behaviours. Our ReactFace generates multiple different but appropriate photo-realistic human facial reactions by: (i) learning an appropriate facial reaction distribution representing multiple different but appropriate facial reactions; and (ii) synchronizing the generated facial reactions with the speaker verbal and non-verbal behaviours at each time stamp, resulting in realistic 2D facial reaction sequences. Experimental results demonstrate the effectiveness of our approach in generating multiple diverse, synchronized, and appropriate facial reactions from each speaker's behaviour. The quality of the generated facial reactions is intimately tied to the speaker's speech and facial expressions, achieved through our novel speaker-listener interaction modules.

In dyadic interactions, humans communicate their intentions and state of mind using verbal and non-verbal cues, where multiple different facial reactions might be appropriate in response to a specific speaker behaviour. Then, how to develop a machine learning (ML) model that can automatically generate multiple appropriate, diverse, realistic and synchronised human facial reactions from an previously unseen speaker behaviour is a challenging task. Following the successful organisation of the first REACT challenge (REACT 2023), this edition of the challenge (REACT 2024) employs a subset used by the previous challenge, which contains segmented 30-secs dyadic interaction clips originally recorded as part of the NOXI and RECOLA datasets, encouraging participants to develop and benchmark Machine Learning (ML) models that can generate multiple appropriate facial reactions (including facial image sequences and their attributes) given an input conversational partner's stimulus under various dyadic video conference scenarios. This paper presents: (i) the guidelines of the REACT 2024 challenge; (ii) the dataset utilized in the challenge; and (iii) the performance of the baseline systems on the two proposed sub-challenges: Offline Multiple Appropriate Facial Reaction Generation and Online Multiple Appropriate Facial Reaction Generation, respectively. The challenge baseline code is publicly available at https://github.com/reactmultimodalchallenge/baseline_react2024.

Robotic well-being coaches have been shown to successfully promote people's mental well-being. To provide successful coaching, a robotic coach should have the capability to repair the mistakes it makes. Past investigations of robot mistakes are limited to game or task-based, one-off and in-lab studies. This paper presents a 4-phase design process to design repair strategies for robotic longitudinal well-being coaching with the involvement of real-world stakeholders: 1) designing repair strategies with a professional well-being coach; 2) a longitudinal study with the involvement of experienced users (i.e., who had already interacted with a robotic coach) to investigate the repair strategies defined in (1); 3) a design workshop with users from the study in (2) to gather their perspectives on the robotic coach's repair strategies; 4) discussing the results obtained in (2) and (3) with the mental well-being professional to reflect on how to design repair strategies for robotic coaching. Our results show that users have different expectations for a robotic coach than a human coach, which influences how repair strategies should be designed. We show that different repair strategies (e.g., apologizing, explaining, or repairing empathically) are appropriate in different scenarios, and that preferences for repair strategies change during longitudinal interactions with the robotic coach.

Irony detection is a complex task that often stumps both humans, who frequently misinterpret ironic statements, and artificial intelligence (AI) systems. While the majority of AI research on irony detection has concentrated on linguistic cues, the role of non-verbal cues like facial expressions and auditory signals has been largely overlooked. This paper investigates the effectiveness of machine learning models in recognizing irony using solely non-verbal cues. To this end, we conducted the following experiments and analysis: (i) we trained and evaluated some machine-learning models to detect irony; (ii) we compared the results with human interpretations; and (iii) we analysed and identified multi-modal non-verbal irony markers. Our research demonstrates that machine learning models trained on nonverbal data have shown significant promise in detecting irony, outperforming human judgments in this task. Specifically, we found that certain facial action units and acoustic characteristics of speech are key indicators of irony expression. These non-verbal cues, often overlooked in traditional irony detection methods, were effectively identified by machine learning models, leading to improved accuracy in detecting irony.

Despite the recent advancements in robotics and machine learning (ML), the deployment of autonomous robots in our everyday lives is still an open challenge. This is due to multiple reasons among which are their frequent mistakes, such as interrupting people or having delayed responses, as well as their limited ability to understand human speech, i.e., failure in tasks like transcribing speech to text. These mistakes may disrupt interactions and negatively influence human perception of these robots. To address this problem, robots need to have the ability to detect human-robot interaction (HRI) failures. The ERR@HRI 2024 challenge tackles this by offering a benchmark multimodal dataset of robot failures during human-robot interactions, encouraging researchers to develop and benchmark multimodal machine learning models to detect these failures. We created a dataset featuring multimodal non-verbal interaction data, including facial, speech, and pose features from video clips of interactions with a robotic coach, annotated with labels indicating the presence or absence of robot mistakes, user awkwardness, and interaction ruptures, allowing for the training and evaluation of predictive models. Challenge participants have been invited to submit their multimodal ML models for detection of robot errors, to be evaluated against various performance metrics such as accuracy, precision, recall, F1 score, with and without a margin of error reflecting the time-sensitivity of these metrics. The results of this challenge will help the research field in better understanding the robot failures in human-robot interactions and designing autonomous robots that can mitigate their own errors after successfully detecting them.

Robotic coaches have been recently investigated to promote mental well-being in various contexts such as workplaces and homes. With the widespread use of Large Language Models (LLMs), HRI researchers are called to consider language appropriateness when using such generated language for robotic mental well-being coaches in the real world. Therefore, this paper presents the first work that investigated the language appropriateness of robot mental well-being coach in the workplace. To this end, we conducted an empirical study that involved 17 employees who interacted over 4 weeks with a robotic mental well-being coach equipped with LLM-based capabilities. After the study, we individually interviewed them and we conducted a focus group of 1.5 hours with 11 of them. The focus group consisted of: i) an ice-breaking activity, ii) evaluation of robotic coach language appropriateness in various scenarios, and iii) listing shoulds and shouldn’ts for designing appropriate robotic coach language for mental well-being. From our qualitative evaluation, we found that a language-appropriate robotic coach should (1) ask deep questions which explore feelings of the coachees, rather than superficial questions, (2) express and show emotional and empathic understanding of the context, and (3) not make any assumptions without clarifying with follow-up questions to avoid bias and stereotyping. These results can inform the design of language-appropriate robotic coach to promote mental well-being in real-world contexts.

In recent years, the awareness of the academy around responsible research has notably increased. For instance, with advances in machine learning and artificial intelligence, recent efforts have been made to promote ethical, fair, and inclusive AI and robotics. To better understand if and to what extent HRI is incentivizing researchers to engage in responsible research, we conducted an exploratory review of the publishing guidelines for the most popular HRI conference venues. We identified 18 conferences which published at least 7 HRI papers in 2022. From these, we discuss four themes relevant to conducting responsible HRI research in line with the Responsible Research and Innovation framework: ethical and human participant considerations, transparency and reproducibility, accessibility and inclusion, and plagiarism and LLM use. We identify several gaps and room for improvement within HRI regarding responsible research. Finally, we establish a call to action to provoke conversations among HRI researchers about the importance of conducting responsible research within emerging fields like HRI.

The last decade has shown a growing interest in robots as well-being coaches. However, insightful guidelines for the design of robots as coaches to promote mental well-being have not yet been proposed. This article details design and ethical recommendations based on a qualitative analysis drawing on a grounded theory approach, which was conducted with a three-step iterative design process which included user-centered design studies involving robotic well-being coaches, namely: (1) a user-centred design study conducted with 11 participants consisting of both prospective users who had participated in a Brief Solution-Focused Practice study with a human coach, as well as coaches of different disciplines, (2) semi-structured individual interview data gathered from 20 participants attending a Positive Psychology intervention study with the robotic well-being coach Pepper, and (3) a user-centred design study conducted with 3 participants of the Positive Psychology study as well as 2 relevant well-being coaches. After conducting a thematic analysis and a qualitative analysis, we collated the data gathered into convergent and divergent themes, and we distilled from those results a set of design guidelines and ethical considerations. Our findings can inform researchers and roboticists on the key aspects to take into account when designing robotic mental well-being coaches.

In recent years, the affective computing research community has put ethics at the centre of its research agenda. However, many of the currently available datasets for affective computing are ‘small’, making bias and debias analysis challenging. This paper presents the first work to explore bias analysis and mitigation of a small temporal multi-modal dataset for mental well-being by adopting different data augmentation techniques. This proof-of-concept work’s contributions include: i) introducing a novel small temporal multi-modal dataset of dyadic interactions during mental well-being coaching; ii) providing multi-modal and feature importance analyses evaluated via modelling performance and fairness metrics across both high and low-level features; and iii) proposing a simple and effective data augmentation strategy (MixFeat) to debias the small dataset presented in this paper. We conduct extensive experiments and analyses to compare our proposed method against other baseline data augmentation method across various uni-modal and multi-modal setups. Our results indicate that, regardless of the dimensionality of the dataset at hand, the inclusion of a bias analysis section in the conference papers is viable. This paper is therefore a call to the community to include a bias analysis section in ACII conference submissions, similar to the ablation studies conducted in papers submitted to major machine learning conferences.

The World Health Organization recommends that employers take action to protect and promote mental well-being at work. However, the extent to which these recommended practices can be implemented in the workplace is limited by the lack of resources and personnel availability. Robots have been shown to have great potential for promoting mental well-being, and the gradual adoption of such assistive technology may allow employers to overcome the aforementioned resource barriers. This paper presents the first study that investigates the deployment and use of two different forms of robotic well-being coaches in the workplace in collaboration with a tech company whose employees (26 coachees) interacted with either a QTrobot (QT ) or a Misty robot (M). We endowed the robots with a coaching personality to deliver positive psychology exercises over four weeks (one exercise per week). Our results show that the robot form significantly impacts coachees' perceptions of the robotic coach in the workplace. Coachees perceived the robotic coach in M more positively than in QT (both in terms of behaviour appropriateness and perceived personality), and they felt more connection with the robotic coach in M. Our study provides valuable insights for robotic well-being coach design and deployment, and contributes to the vision of taking robotic coaches into the real world.

The Multiple Appropriate Facial Reaction Generation Challenge (REACT2023) is the first competition event focused on evaluating multimedia processing and machine learning techniques for generating human-appropriate facial reactions in various dyadic interaction scenarios, with all participants competing strictly under the same conditions. The goal of the challenge is to provide the first benchmark test set for multi-modal information processing and to foster collaboration among the audio, visual, and audio-visual behaviour analysis and behaviour generation (a.k.a generative AI) communities, to compare the relative merits of the approaches to automatic appropriate facial reaction generation under different spontaneous dyadic interaction conditions. This paper presents: (i) the novelties, contributions and guidelines of the REACT2023 challenge; (ii) the dataset utilized in the challenge; and (iii) the performance of the baseline systems on the two proposed sub-challenges: Offline Multiple Appropriate Facial Reaction Generation and Online Multiple Appropriate Facial Reaction Generation, respectively. The challenge baseline code is publicly available at https://github.com/reactmultimodalchallenge/baseline_react2023.

With the introduction of socially assistive robots in many avenues of children’s lives, it is becoming increasingly vital to understand how children’s perceptions of the robot affect their evaluation and interaction. The main objective of this work is to investigate how children’s anxiety towards robots has influenced their perceptions of their interaction with a Nao robot. We collected data from 37 children (8 - 13 years old) who interacted, for about 30-45 minutes, with the robot which delivered initial pleasantries and four different tasks to help assess their mental wellbeing in a lab setting. We collected audio-visual recordings of the interaction. At the end of the session, we asked children to answer three self-report questionnaires to evaluate: the robot’s role as a confidante, the anxiety towards the robot, and the children’s perception of the robot’s behaviour and capabilities. Based on their responses to the robot’s anxiety questionnaire, children were divided into two categories: “low anxiety” (anxiety scoremedian anxiety score). Our results show that i) most children (89.2%) irrespective of their wellbeing, experience some degree of anxiety towards the robot, ii) children’s anxiety has influenced their willingness to participate in the initial pleasantries conducted by the robot, and iii) children’s anxiety has also affected their evaluations of the robot as a confidante and their perceptions of the robot’s behaviour and capabilities. Findings from this work have significant implications for designing effective and successful robot-led initiatives for assessing mental wellbeing in children, by taking into account their mindsets and dispositions.

Robotic mental well-being coaches could be used to help people maintain their well-being, and improve access to mental healthcare. In coaching, the alliance between the coach and coachee is important for the success of the practice. However, this alliance might be negatively affected by interaction ruptures (e.g., the robot making mistakes and the user feeling awkward) that still commonly occur in human-robot interactions. Therefore, robotic coaches should be able to recognize ruptures occurring during their interactions with human users to guarantee the success of the well-being practice. To this aim, we analyse coachee behavioural responses to interaction ruptures during a robotic positive psychology coaching practice and how these behavioural cues evolve over time. We focus our analysis on a dataset we collected in a previous work, where 26 participants interacted with either a QTrobot or a Misty II robot at their workplace over 4 weeks. We undertake a longitudinal analysis of coachees’ multimodal nonverbal cues (i.e., facial expressions, vocal acoustic features, and body pose features) to investigate the contribution of individual modalities for detecting interaction ruptures. Our results show that coachees: i) displayed facial cues of rupture (e.g, laughing at the robot) and suspicion more in the first week than in the last week; ii) talked more and were less silent in the last week than in the previous weeks; and iii) exhibited a higher number of hand-over-face gestures (a cue for self-disclosure) in the last week than in the previous weeks. Our findings aim to inform the development of AI models for multi-modal detection of interaction ruptures which can be used to improve the effectiveness and the success of robotic well-being coaching.

Neurodevelopmental disorders (NDDs) are charac-terised by impairments in communication. Socially assistive robots have been identified as a promising avenue to alleviate their burden. Since NDDs have different needs, their way of communicating with a robot (e.g., speech-based) could differ among individuals. This paper aims to investigate the most suitable modality to communicate with a robot for NDDs - among voice, cards, and buttons - and explore their opinion on this matter. We ran an exploratory study involving 29 NDDs participants: 13 of them could freely communicate with an autonomous QT robot, 9 took part in a group discussion, and 7 first interacted individually with the robot, and then they participated in a group discussion. Our results showed that i) the cards were the most used communication modality, ii) voice can be used for counting games, buttons for multiple-choice games, and cards for memory-like games, iii) opinions did not differ much among groups.*

Despite the growing interest in smart homes and robotics in many domains, very few studies have explored how socially assistive robots (SAR) can be integrated into smart homes to control them while socially interacting with people. This paper explores two factors - embodiment and movement - that influence the human-robot interaction into a domestic context. We conducted a within-subjects study with three con-ditions (disembodied-static, embodied-static, embodied-dynamic) involving the conventional population. Participants (N = 10) interacted into two speech-based tasks with an autonomous Temi robot fully integrated with a smart home (e.g., lights, room temperature, music, oven control) and answered questions about their perception towards the robot, including perceived sociability and social presence. The results indicated that participants perceived the embodied-movable robot as significantly more sociable and socially present than static or disembodied ones.

Socially Assistive Robots (SARs) show promise in helping children during therapeutic and clinical interventions. However, using SARs for the evaluation of mental wellbeing of children has not yet been explored. Thus, this paper presents an empirical study with 28 children 8-13 years old interacting with a Nao robot in a 45-minute session where the robot administered (robotised) the Short Mood and Feelings Questionnaire (SMFQ) and the Revised Child Anxiety and Depression Scale (RCADS). Prior to the experimental session, we also evaluated children’s wellbeing using established standardised approaches via online RCADS questionnaires filled by the children (self-report) and their parents (parent-report). We clustered the participants into three groups (lower, medium, and higher tertile) based on their SMFQ scores. Further, we analysed the questionnaire responses across the three clusters and across the different modes of administration (self-report, parent-report, and robotised). Our results show that the robotised evaluation seems to be the most suitable mode in identifying wellbeing related anomalies in children across the three clusters of participants as compared with the self-report and the parent-report modes. Further, children with decreasing levels of wellbeing (lower, medium and higher tertiles) exhibit different response patterns: children of higher tertile are more negative in their responses to the robot while the ones of lower tertile are more positive in their responses to the robot. Findings from this work show that SARs can be a promising tool to potentially evaluate mental wellbeing related concerns in children.

Empathy is the ability to share someone else’s feelings or experiences; it influences how people interact and relate. Socially assistive robots (SAR) are a promising means of conveying and eliciting empathy toward facilitating human-robot interaction. This work examines factors that influence the amount of empathy elicited by a SAR storyteller and users’ perceptions of that robot. We conducted an empirical mixed-design study (N=46) using an autonomous SAR storyteller that told three stories, each with a different human or robot target of empathy. The robot storyteller used the first-person narrative voice (1PNV) with half of the participants and the third-person narrative voice (3PNV) with the other half. We found that the SAR storyteller elicited significantly more empathy when the story target of empathy matched the SAR narrator, i.e., was also a robot. Additionally, the 1PNV robot elicited significantly more empathy and was perceived as more human-like, easy to interact with, and trustworthy than the 3PNV robot. Finally, participants who empathized more with the robot displayed facial expressions consistent with the emotional story content. These insights inform the design of SAR storytellers capable of eliciting empathy toward creating compelling and effective human-robot interactions.

Virtual and robotic agents capable of perceiving human empathy have the potential to participate in engaging and meaningful human-machine interactions that support human well-being. Prior research in computational empathy has focused on designing empathic agents that use verbal and nonverbal behaviors to simulate empathy and attempt to elicit empathic responses from humans. The challenge of developing agents with the ability to automatically perceive elicited empathy in humans remains largely unexplored. Our paper presents the first approach to modeling user empathy elicited during interactions with a robotic agent. We collected a new dataset from the novel interaction context of participants listening to a robot storyteller (46 participants, 6.9 hours of video). After each storytelling interaction, participants answered a questionnaire that assessed their level of elicited empathy during the interaction with the robot. We conducted experiments with 8 classical machine learning models and 2 deep learning models (long short-term memory networks and temporal convolutional networks) to detect empathy by leveraging patterns in participants’ visual behaviors while they were listening to the robot storyteller. Our highest-performing approach, based on XGBoost, achieved an accuracy of 69% and AUC of 72% when detecting empathy in videos. We contribute insights regarding modeling approaches and visual features for automated empathy detection. Our research informs and motivates future development of empathy perception models that can be leveraged by virtual and robotic agents during human-machine interactions.

The research and development of socially interactive robots is a complex challenge because of the wide variety of capabilities needed for effective social human-robot interactions (HRI). Many of these capabilities, including perception, dialog, and control, have state of the art methods and solutions, but combining those into a comprehensive and seamless interaction is still an open challenge. We describe HARMONI, a multi-modal, open-source tool for rapid social HRI development and deployment. HARMONI is centered around a ROS package for interaction development, including decision management and node orchestration. HARMONI systematically integrates with disparate functionalities needed to conduct a meaningful social human-robot interaction such as external cloud services, AI models, and modules for sensing, planning, and acting on a variety of platforms. HARMONI was applied to the QT robot platform and usability tests were conducted to evaluate the ease and speed of development and deployment. This paper describes the architecture and design of HARMONI and reports the results of a pilot study with novice users.

Abstract Phygital interaction is a form of tangible interaction where digital and physical contents are combined in such a way that the locus of multimedia information is detached from the physical material(s) manipulated by the user. The use of phygital interaction is supported by several theoretical approaches that emphasize the development of cognitive skills dependent upon embodied interactions with the physical environment. Several studies demonstrate the potential of using phygital technologies for supporting people with intellectual disabilities (ID) in the development of cognitive, sensorimotor, social and behavioral skills. Our research aims at exploring the potential of phygital interaction for (young) adults with ID in a real setting, using a research platform called Reflex as a case study. For this purpose, we ran an empirical study involving 17 participants with ID and 8 specialists, and compared Reflex with approaches making use of only digital contents or paper-based materials. Our findings highlighted the potentials of phygital approaches to perform interventions with people with ID, enhancing their performances with an appreciated interaction method. In addition, the post-study interviews with specialists favoured the adoption of phygital technologies in a social care context.

The dramatic increment of communication impairments among children increases the demand for intensive, highly accessible and low-cost interventions as well as new assessment and therapeutic tools. Our research aims at exploring the use of Conversational Agents (CAs) to support linguistic assessment and training among children with language impairment. One of the open research issues in this arena concerns the identification of the most appropriate form of "embodiment" of the CA for children to interact with. To this end, we evaluated the linguistic performance of 14 neuro-typical children and 3 children with language impairment comparing different CAs - physical object and virtual character - with "traditional" human interaction. Based on our analysis, we identify insights for the design of CA: the physicality does influence the performance of linguistic tasks for children with linguistic impairment. In addition, children seem to show a preference for the physical CA and perceived it as smarter than the virtual one.