Participatory design effectively engages stakeholders in technology development but is often constrained by small, resource-intensive activities. This study explores a scalable complementary method, enabling broad pattern identification in the design for interfaces in autonomous vehicles. We implemented a human-centered, iterative process that combined crowd creativity, structured participatory principles, and expert feedback. Across iterations, participant concepts evolved from simple cues to multimodal systems. Novel suggestions ranged from personalized features, like tracking lights, to inclusive elements like haptic feedback, progressively refining designs toward greater contextual awareness. To assess outcomes, we compared representative designs: a popular-design, reflecting the most frequently proposed ideas, and an innovative-design, merging participant innovations with expert input. Both were evaluated against a benchmark through video-based simulations. Results show that the popular-design outperformed the alternatives on both interpretability and user experience, with expert-validated innovations performing second best. These findings highlight the potential of scalable participatory methods for shaping emerging technologies.

Robot malfunctions are unavoidable in human–robot collaboration and oftentimes detrimental. Yet humans are rarely instructed on how to respond in such moments, leaving ample room for spontaneity and unpredictability. We studied 65 participants working alongside a collaborative robot under both normal operation and deliberate malfunction conditions. We analyzed unscripted vocal and action responses regarding situational awareness (SA)—whether malfunctions were noticed—and task-oriented response appropriateness—whether responses advanced or undermined the collaboration. During malfunctions, SA was universal, as was frustration and confusion, yet appropriateness diverged sharply: unscripted responses ranged from clarifying questions and corrective actions to sarcasm, comedic gestures, and intentional mismarkings. Efficiency malfunctions elicited far more productive responses than effectiveness malfunctions did, underscoring how actionability fundamentally shapes human intervention. Our findings reveal a fragile link between SA and task-aligned action, highlighting the need for robot transparency, explainability and adaptability, so collaborators are actively supported when things fail.

We present the concept of the auxthetic mind-body (AM): a system which extends the human mind and body to include robot bodies, artificial “thoughts,” and artificial feelings as part of the perception of “self.” While human-robot interaction research has long grappled with embodiment, the AM represents an as-yet unexplored space in this realm and raises a host of questions around its uses, consequences, and preservation of human agency. We explore the concept through speculative sociotechnical design, foregrounding how the technology might make us feel (rather than what it might do) as a guiding foundation for further development. Through poetry and marginalia, we invite readers to reflect on what it might mean to think, feel, and be with an AM. In doing so, we sketch both technical possibility and future worth longing for—one where the dissolution of the human-machine boundary can be meaningful, grounded, and less frightening than it may seem.

Perinatal depression (PND) is a common mental health disorder associated with childbirth, which has high societal costs affecting up to 10% of individuals during pregnancy or postpartum. Whilst socially assistive robots (SARs) have recently proven to be useful tools in mental healthcare, and our previous work has investigated different stakeholders’ perspectives on SARs in PND screening through interview studies, gaps remain in understanding how primary users (i.e., prospective patients) perceive and interact with such technologies. In this article, we use a participatory design methodology with semi-structured interviews of women in Sweden with previous experience of PND to explore the roles that SARs could play in addressing PND challenges and identify design factors for SARs in PND screening. We design and evaluate in a user study a robot prototype in two new interaction contexts for SARs with different levels of human oversight. The results show that SARs are welcomed by most participants, who appreciated the potentially faster assessment process and felt more comfortable opening up with a robot versus a human clinician. However, we found that there is no single solution that fits all, as other participants preferred the flexibility of self-reported digital surveys or interaction with a human clinician. Moreover, results show that transparency and human oversight are crucial requirements to consider when implementing robot-delivered PND screening questionnaires and diagnostic interviews. We reflect on ethical considerations, provide design recommendations and urge HRI designers to carefully consider whom SARs benefit, whom they may not, and which safeguarding factors are necessary to prevent potential negative outcomes.

As autonomous vehicles become more integrated into shared human environments, effective communication with road users is essential for ensuring safety. While previous research has focused on developing external Human-Machine Interfaces (eHMIs) to facilitate these interactions, we argue that involving users in the early creative stages can help address key challenges in the development of this technology. To explore this, our study adopts a participatory, crowd-sourced approach to gather user-generated ideas for eHMI designs. Participants were first introduced to fundamental eHMI concepts, equipping them to sketch their own design ideas in response to scenarios with varying levels of perceived risk. An initial pre-study with 29 participants showed that while they actively engaged in the process, there was a need to refine task objectives and encourage deeper reflection. To address these challenges, a follow-up study with 50 participants was conducted. The results revealed a strong preference for autonomous vehicles to communicate their awareness and intentions using lights (LEDs and projections), symbols, and text. Participants’ sketches prioritized multi-modal communication, directionality, and adaptability to enhance clarity, consistently integrating familiar vehicle elements to improve intuitiveness.

Current robotic agents, such as autonomous vehicles (AVs) and drones, need to deal with uncertain real-world environments with appropriate situational awareness (SA), risk awareness, coordination, and decision-making. The SymAware project strives to address this issue by designing an architecture for artificial awareness in multi-agent systems, enabling safe collaboration of autonomous vehicles and drones. However, these agents will also need to interact with human users (drivers, pedestrians, drone operators), which in turn requires an understanding of how to model the human in the interaction scenario, and how to foster trust and transparency between the agent and the human.In this work, we aim to create a data-driven model of a human driver to be integrated into our SA architecture, grounding our research in the principles of trustworthy human-agent interaction. To collect the data necessary for creating the model, we conducted a large-scale user-centered study on human-AV interaction, in which we investigate the interaction between the AV’s transparency and the users’ behavior.The contributions of this paper are twofold: First, we illustrate in detail our human-AV study and its findings, and second we present the resulting Markov chain models of the human driver computed from the study’s data. Our results show that depending on the AV’s transparency, the scenario’s environment, and the users’ demographics, we can obtain significant differences in the model’s transitions.

Advancements in robotic capabilities for providing physical assistance, psychological support, and daily health management are making the deployment of intelligent health-care robots in home environments increasingly feasible in the near future. However, challenges arise when the information provided by these robots contradicts users’ memory, raising concerns about user trust and decision-making. This paper presents a study that examines how varying a robot’s level of transparency and sociability influences user interpretation, decision-making and perceived trust when faced with conflicting information from a robot. In a 2 × 2 between-subjects online study, 176 participants watched videos of a Furhat robot acting as a family healthcare assistant and suggesting a fictional user to take medication at a different time from that remembered by the user. Results indicate that robot transparency influenced users’ interpretation of information discrepancies: with a low transparency robot, the most frequent assumption was that the user had not correctly remembered the time, while with the high transparency robot, participants were more likely to attribute the discrepancy to external factors, such as a partner or another household member modifying the robot’s information. Additionally, participants exhibited a tendency toward overtrust, often prioritizing the robot’s recommendations over the user’s memory, even when suspecting system malfunctions or third-party interference. These findings highlight the impact of transparency mechanisms in robotic systems, the complexity and importance associated with system access control for multi-user robots deployed in home environments, and the potential risks of users’ over-reliance on robots in sensitive domains such as healthcare.

While there is evidence that human-like characteristics in robots could benefit child-robot interaction in many ways, open questions remain about the appropriate degree of human likeness that should be implemented in robots to avoid adverse effects on acceptance and trust. This study investigates how human likeness, appearance and behavior, influence children’s social and competency trust in a robot. We first designed two versions of the Furhat robot with visual and auditory human-like and machine-like cues validated in two online studies. Secondly, we created verbal behaviors where human likeness was manipulated as responsiveness regarding the robot’s lexical matching. Then, 52 children (7–10 years old) played a storytelling game in a between-subjects experimental design. Results show that the conditions did not affect subjective trust measures. However, objective measures showed that human likeness affects trust differently. While low human-like appearance enhanced social trust, high human-like behavior improved children’s acceptance of the robot’s task-related suggestions. This work provides empirical evidence on manipulating facial features and behavior to control human likeness in a robot with a highly human-like morphology. We discuss the implications and importance of balancing human likeness in robot design and its impacts on task performance, as it directly impacts trust-building with children.

In developing a computational model of trust, this paper summarises the findings in a previous study exploring mutual gaze as a behavioural parameter of social trust and liking [1]. Drawing on the data collected in a related paper [6], which provides us with video clips of children interacting with a robot during a collaborative storytelling game, we look at the interactions between metrics assessing social trust and liking, and the development of mutual gaze as an objective measure of social trust and liking. We achieve this through several statistical analyses between the percent of mutual gaze in each interaction, scores from social trust and liking metrics, age of the participant, and duration. The findings of our study support the use of mutual gaze as an objective measure for liking, but there is still not sufficient evidence to support the use of mutual gaze as an objective measure to identify and capture social trust as a whole. Furthermore, interaction context impacts the amount of mutual gaze in an interaction, and the age of the participant has an impact on the amount of mutual gaze that occurs.

While creativity has been previously studied in Child-Robot Interaction (cHRI), the effect of regulatory focus on creativity skills has not been investigated. This paper presents an exploratory study that, for the first time, uses the Regulatory Focus Theory (RFT) to assess children's creativity skills in an educational context with a social robot. We investigated whether two key emotional regulation techniques, promotion (approach) and prevention (avoidance), stimulate creativity during a story-telling activity between a child and a robot. We conducted a between-subjects field study with 69 children between the ages of 7 and 9 years old, divided between two study conditions: (1) promotion, where a social robot primes children for action by eliciting positive emotional states, and (2) prevention, where a social robot primes children for avoidance by evoking a states related to security and safety associated with blockage-oriented behaviors. To assess changes in creativity as a response to the priming interaction, children were asked to tell stories to the robot before (pre-test) and after (post-test) the priming interaction. We measured creativity levels by analyzing the verbal content of the stories. We coded verbal expressions related to creativity variables, including fluency, flexibility, elaboration, and originality. Our results show that children in the promotion condition generated significantly more ideas, and their ideas were on average more original in the stories they created in the post-test rather than in the pre-test. We also modeled the process of creativity that emerges during storytelling in response to the robot's verbal behavior. This paper enriches the scientific understanding of creativity emergence in child-robot collaborative interactions.

In this paper we investigate the possibility of socially assistive robots (SARs) supporting diagnostic screening for peripartum depression (PPD) within the next five years. Through a HRI/socio-legal collaboration, we explore the gender norms within PPD in Sweden, to inform a gender-sensitive approach to designing SARs in such a setting, as well as governance implications. This is achieved through conducting expert interviews and qualitatively analysing the data. Based on the results, we conclude that a gender-sensitive approach is a necessity in relation to the design and governance of SARs for PPD screening.

Infant engagement during guided play is a reliable indicator of early learning outcomes, psychiatric issues and familial wellbeing. An obstacle to using such information in real-world scenarios is the need for a domain expert to assess the data. We show that an end-to-end Deep Learning approach can perform well in automatic infant engagement detection from a single video source, without requiring a clear view of the face or the whole body. To tackle the problem of explainability in learning methods, we evaluate how four common attention mapping techniques can be used to perform subjective evaluation of the network’s decision process and identify multimodal cues used by the network to discriminate engagement levels. We further propose a quantitative comparison approach, by collecting a human attention baseline and evaluating its similarity to each technique.

Antenatal depression impacts 7-20% of women globally, and can have serious consequences for both the mother and the infant. Preventative interventions are effective, but are cost-efficient only among those at high risk. As such, being able to predict and identify those at risk is invaluable for reducing the burden of care and adverse consequences, as well as improving treatment outcomes. While several approaches have been proposed in the literature for the automatic prediction of depressive states, there is a scarcity of research on automatic prediction of perinatal depression. Moreover, while there exist some works on the automatic prediction of postpartum depression using data collected in clinical settings and applied the model to a smartphone application, to the best of our knowledge, no previous work has investigated the automatic prediction of late antenatal depression using data collected via a smartphone app in the first and second trimesters of pregnancy. This study utilizes data measuring various aspects of self-reported psychological, physiological and behavioral information, collected from 915 women in the first and second trimester of pregnancy using a smartphone app designed for perinatal depression. By applying machine learning algorithms on these data, this paper explores the possibility of automatic early detection of antenatal depression (i.e., during week 36 to week 42 of pregnancy) in everyday life without the administration of healthcare professionals. We compare uni-modal and multi-modal models and identify predictive markers related to antenatal depression. With multi-modal approach the model reaches a BAC of 0.75, and an AUC of 0.82.

In the context of child development, practice is recognised as one of the essential activities to stimulate creativity. Here we aimed to explore whether repeated interactions with a virtual social robot could help build up children's creative performance over time. To this end, we developed an interactive storytelling game with the virtual robot Furhat. Twenty-five children between 9- and 12- years old played the online game two times with seven days of zero exposure in between. Our results revealed that repeated encounters have mixed effects on verbal creativity: while children were more creative in terms of flexibility, fluency, and elaboration in the second interaction, the level of originality remained stagnant. Moreover, the second encounter positively affected children's collaboration with and social behaviour toward the virtual robot. These results provide valuable evidence supporting the potential of multiple interactions with artificial agents to foster children's creativity over time. This paper, thus, provides readers with (1) a novel approach to stimulating verbal creativity through practice with artificial agents, (2) an assessment of the creative process in repeated interactions, and (3) evidence of how the behaviour of the robot influences children's creativity and their behaviour over time.

Studies in Child-Robot Interaction have shown that children form first impressions of a robot’s trustworthiness that might influence how they interact with social robots in long-term interactions. However, how children’s trust in robots evolves and how it relates to relationship formation is not well understood. This study investigates the effects of repeated encounters with a virtual social robot on children’s social and competency trust in social robots and their relationship formation. We developed an online storytelling game with the Furhat robot, where 25 children (9-12 years old) played with the robot over two sessions with seven days of zero exposure in between. Results show that children’s competency trust improved with time. We also found empirical evidence that children felt closer to the robot in the second encounter. This work enriches the scientific understanding of children’s trust development in social robots over extended periods of time in child-robot collaborative interactions.

We present SLOT-V, a novel supervised learning framework that learns observer models (human preferences) from robot motion trajectories in a legibility context. Legibility measures how easily a (human) observer can infer the robot’s goal from a robot motion trajectory. When generating such trajectories, existing planners often rely on an observer model that estimates the quality of trajectory candidates. These observer models are frequently hand-crafted or, occasionally, learned from demonstrations. Here, we propose to learn them in a supervised manner using the same data format that is frequently used during the evaluation of aforementioned approaches. We then demonstrate the generality of SLOT-V using a Franka Emika in a simulated manipulation environment. For this, we show that it can learn to closely predict various hand-crafted observer models, i.e., that SLOT-V’s hypothesis space encompasses existing handcrafted models. Next, we showcase SLOT-V’s ability to generalize by showing that a trained model continues to perform well in environments with unseen goal configurations and/or goal counts. Finally, we benchmark SLOT-V’s sample efficiency (and performance) against an existing IRL approach and show that SLOT-V learns better observer models with less data. Combined, these results suggest that SLOT-V can learn viable observer models. Better observer models imply more legible trajectories, which may -in turn - lead to better and more transparent human-robot interaction.

Understanding the way different robot’s strategies affect children’s perceptions of social robots is crucial for a trustworthy child-robot relationship. This paper presents a preliminary study on whether motivational strategies based on Regulatory Focus Theory and goal attainment affect children’s perception of a virtual social robot when solving a task. The ongoing pandemic (COVID-19) is altering the way we perform research. Hence, we designed a fully autonomous game with a virtual social robot. In an online user study, 25 children (8 to 17 years old) played a regulatory focus goal-oriented game with a virtual child-like version of the Furhat robot. We evaluated children’s perceptions of the robot’s social trust, competency trust, and likability. Also, we assessed the children’s affective state (valence and arousal) before and after playing the game. Our preliminary results show that in the prevention condition, fulfilling the goal elicited less happiness in children. Surprisingly, we observed a trend increase in the social and competency trust elicited by the virtual robot when children were prevented from fulfilling the goal of the task. We discuss the results and the effects of online setups on conducting user studies with children.

According to psychology research, emotional induction has positive implications in many domains such as therapy and education. Our aim in this paper was to manipulate the Regulatory Focus Theory to assess its impact on the induction of regulatory focus related emotions in children in a pretend play scenario with a social robot. The Regulatory Focus Theory suggests that people follow one of two paradigms while attempting to achieve a goal; by seeking gains (promotion focus - associated with feelings of happiness) or by avoiding losses (prevention focus - associated with feelings of fear). We conducted a study with 69 school children in two different conditions (promotion vs. prevention). We succeeded in inducing happiness emotions in the promotion condition and found a resulting positive effect of the induction on children’s social engagement with the robot. We also discuss the important implications of these results in both educational and child robot interaction fields.

The Internet of Medical Things (IoMT) is an extension and specialization of that original Internet of Things (IoT) concept, and applies to the interconnectedness of devices, software applications, and data which are specific to the medical industry. IoMT can add smart technologies to medical devices to monitor the progression of a disease away from the doctor’s office and learn things that could impact future care guidelines and patients. It can also provide a better way to care for our elderly by tracking vitals and heart performance, glucose and other body systems, and activity and sleeping levels. During the outbreak of pandemic (e.g., COVID-19), IoMT can even be used to detect main symptoms ubiquitously using intelligent sensors and trace the origin of the outbreak based on aggregated IoT data (e.g., geographic mobile data and purchase history). Although most of the contemporary IoMT systems can measure risks, make decisions, and take actions automatically, the lack of emotion-aware abilities will be an obstacle to more harmonious human–machine interaction and more efficient medical process. Besides, mental disorders, such as depression, schizophrenia, and anxiety, have become a more noticeable cause of suffering. The integration of emotion-aware abilities into IoMT can also contribute to monitor emotional dysregulation continuously in subjects with mental disorders or undergoing serious pandemic such as COVID-19, and give these patients personalized therapy recommendations. Research on affective computing has defined a framework to recognize, interpret, and process human affects, but more research is needed to investigate its application to biomedical applications, especially “in the wild” and over extended periods of time, and how to integrate emotion-aware abilities into IoMT organically is still an open question. This special issue aims to create a platform for researchers, developers, and practitioners from both academia and industry to disseminate the state-of-the-art results and to advance the Emotion-Aware ubiquitous computing in IoMT.

The issue of how to make embodied agents explainable has experienced a surge of interest over the past 3 years, and there are many terms that refer to this concept, such as transparency and legibility. One reason for this high variance in terminology is the unique array of social cues that embodied agents can access in contrast to that accessed by non-embodied agents. Another reason is that different authors use these terms in different ways. Hence, we review the existing literature on explainability and organize it by (1) providing an overview of existing definitions, (2) showing how explainability is implemented and how it exploits different social cues, and (3) showing how the impact of explainability is measured. Additionally, we present a list of open questions and challenges that highlight areas that require further investigation by the community. This provides the interested reader with an overview of the current state of the art.

Today we are witnessing an increased robotisation in all areas of society, from manufacturing to assistive technology, from healthcare to education. These application areas require robots to be able to interact with humans in an efficient and socially acceptable manner. At the same time, like all technologies, robots may not only bring benefits, but also change how we think and behave. This calls for human-robot interaction researchers to design and develop more human-centric and trustworthy artificial intelligence and robotics, which put humans at the center and preserve human agency and autonomy, where robots adapt to the way humans communicate in the world, rather than the other way around. My research career has been devoted to the questions surrounding whether and how machines should embody human-like qualities, in order to be truly human-centric and trustworthy. In this talk I will present examples of human-robot interaction studies and technical advances in socially assistive scenarios from my research at the Uppsala Social Robotics Lab [1], investigating different levels of human-like qualities in robots, from appearance to behaviours and their synergies, in the quest for more human-centric robots.

Numerous studies in social psychology have shown that familiarization across repeated interactions improves people's perception of the other. If and how these findings relate to human-robot interaction (HRI) is not well understood, even though such knowledge is crucial when pursuing long-term interactions. In our work, we investigate the persistence of first impressions by asking 49 participants to play a geography game with a robot. We measure how their perception of the robot changes over three sessions with three to ten days of zero exposure in between. Our results show that different perceptual dimensions stabilize within different time frames, with the robot's competence being the fastest to stabilize and perceived threat the most fluctuating over time. We also found evidence that perceptual differences between robots with varying levels of humanlikeness persist across repeated interactions. This study has important implications for HRI design as it sheds new light on the influence of robots' embodiment and interaction abilities. Moreover, it also impacts HRI theory as it presents novel findings contributing to research on the uncanny valley and robot perception in general.

Mind perception in robots has been an understudied construct in human-robot interaction (HRI) compared to similar concepts such as anthropomorphism and the intentional stance. In a series of three experiments, we identify two factors that could potentially influence mind perception and moral concern in robots: how the robot is introduced (framing), and how the robot acts (social behaviour). In the first two online experiments, we show that both framing and behaviour independently influence participants' mind perception. However, when we combined both variables in the following real-world experiment, these effects failed to replicate. We hence identify a third factor post-hoc: the online versus real-world nature of the interactions. After analysing potential confounds, we tentatively suggest that mind perception is harder to influence in real-world experiments, as manipulations are harder to isolate compared to virtual experiments, which only provide a slice of the interaction.

Facial expressions of emotions influence the perception of robots in first encounters. People can judge trustworthiness, likability, and aggressiveness in a few milliseconds by simply observing other individuals' faces. While first impressions have been extensively studied in adult-robot interaction, they have been addressed in child-robot interaction only rarely. This knowledge is crucial, as the first impression children build of robots might influence their willingness to interact with them over extended periods of time, for example in applications where robots play the role of companions or tutors. The present study focuses on investigating the effects of facial expressions of emotions on children's perceptions of trust towards robots during first encounters. We constructed a set of facial expressions of happiness and anger varying in terms of intensity. We implemented these facial expressions onto a Furhat robot that was either male-like or female-like. 129 children were exposed to the robot's expressions for a few seconds. We asked them to evaluate the robot in terms of trustworthiness, likability, and competence and investigated how emotion type, emotion intensity, and gender-likeness affected the perception of the robot. Results showed that a few seconds are enough for children to make a trait inference based on the robot's emotion. We observed that emotion type, emotion intensity, and gender-likeness did not directly affect trust, but the perception of likability and competence of the robot served as facilitator to judge trustworthiness.

With an ever increasing demand for personal service robots and artificial assistants, companies, start-ups and researchers aim to better understand what makes robot platforms more likable. Some argue that increasing a robot's humanlikeness leads to a higher acceptability. Others, however, find that extremely humanlike robots are perceived as uncanny and are consequently often rejected by users. When investigating people's perception of robots, the focus of the related work lies almost solely on the first impression of these robots, often measured based on images or video clips of the robots alone. Little is known about whether these initial positive or negative feelings persist when giving people the chance to interact with the robot. In this paper, 48 participants were gradually exposed to the capabilities of a robot and their perception of it was tracked from their first impression to after playing a short interactive game with it. We found that initial uncanny feelings towards the robot were significantly decreased after getting to know it better, which further highlights the importance of using real interactive scenarios when studying people's perception of robots. In order to elicit uncanny feelings, we used the 3D blended embodiment Furhat and designed four different facial textures for it. Our work shows that a blended platform can cause different levels of discomfort towards it depending on the facial texture and may thus be an interesting tool for further research on the uncanny valley.

In socially assistive robotics, an important research area is the development of adaptation techniques and their effect on human-robot interaction. We present a meta-learning based policy gradient method for addressing the problem of adaptation in human-robot interaction and also investigate its role as a mechanism for trust modelling. By building an escape room scenario in mixed reality with a robot, we test our hypothesis that bi-directional trust can be influenced by different adaptation algorithms. We found that our proposed model increased the perceived trustworthiness of the robot and influenced the dynamics of gaining human's trust. Additionally, participants evaluated that the robot perceived them as more trustworthy during the interactions with the meta-learning based adaptation compared to the previously studied statistical adaptation model.

Deep reinforcement learning has recently been widely applied in robotics to study tasks such as locomotion and grasping, but its application to social human-robot interaction (HRI) remains a challenge. In this paper, we present a deep learning scheme that acquires a prior model of robot approaching behavior in simulation and applies it to real-world interaction with a physical robot approaching groups of humans. The scheme, which we refer to as Staged Social Behavior Learning (SSBL), considers different stages of learning in social scenarios. We learn robot approaching behaviors towards small groups in simulation and evaluate the performance of the model using objective and subjective measures in a perceptual study and a HRI user study with human participants. Results show that our model generates more socially appropriate behavior compared to a state-of-the-art model.

This work explores a group learning scenario with an autonomous empathic robot. We address two research questions: (1) Can an autonomous robot designed with empathic competencies foster collaborative learning in a group context? (2) Can an empathic robot sustain positive educational outcomes in long-term collaborative learning interactions with groups of students? To answer these questions, we developed an autonomous robot with empathic competencies that is able to interact with a group of students in a learning activity about sustainable development. Two studies were conducted. The first study compares learning outcomes in children across three conditions: learning with an empathic robot; learning with a robot without empathic capabilities; and learning without a robot. The results show that the autonomous robot with empathy fosters meaningful discussions about sustainability, which is a learning outcome in sustainability education. The second study features groups of students who interact with the robot in a school classroom for 2 months. The long-term educational interaction did not seem to provide significant learning gains, although there was a change in game-actions to achieve more sustainability during game-play. This result reflects the need to perform more long-term research in the field of educational robots for group learning.

To feel novel and engaging over time it is critical for an autonomous agent to have a large corpus of potential responses. As the size and multi-domain nature of the corpus grows, however, traditional hand-authoring of dialogue content is no longer practical. While crowdsourcing can help to overcome the problem of scale, a diverse set of authors contributing independently to an agent's language can also introduce inconsistencies in expressed behavior. In terms of affect or mood, for example, incremental authoring can result in an agent who reacts calmly at one moment but impatiently moments later with no clear reason for the transition. In contrast, affect in natural conversation develops over time based on both the agent's personality and contextual triggers. To better achieve this dynamic, an autonomous agent needs to (a) have content and behavior available for different desired affective states and (b) be able to predict what affective state will be perceived by a person for a given behavior. In this proof-of-concept paper, we explore a way to elicit and evaluate affective behavior using crowdsourcing. We show that untrained crowd workers are able to author content for a broad variety of target affect states when given semi-situated narratives as prompts. We also demonstrate that it is possible to strategically combine multimodal affective behavior and voice content from the authored pieces using a predictive model of how the expressed behavior will be perceived.

In the domain of robotic tutors, personalised tutoring has started to receive scientists' attention, but is still relatively underexplored. Previous work using reinforcement learning (RL) has addressed personalised tutoring from the perspective of affective policy learning. However, little is known about the effects of robot behaviour personalisation on user's task performance. Moreover, it is also unclear if and when personalisation may be more beneficial than a robot that adapts to its users and the context of the interaction without personalising its behaviour. In this paper we build on previous work on affective policy learning that used RL to learn what robot's supportive behaviours are preferred by users in an educational scenario. We build a RL framework for personalisation that allows a robot to select verbal supportive behaviours to maximise the user's task progress and positive reactions in a learning scenario where a Pepper robot acts as a tutor and helps people to learn how to solve grid-based logic puzzles. A between-subjects design user study showed that participants were more efficient at solving logic puzzles and preferred a robot that exhibits more varied behaviours compared with a robot that personalises its behaviour by converging on a specific one over time. We discuss insights on negative effects of personalisation and report lessons learned together with design implications for personalised robots.

In this paper, we investigate the applicability of deep learning methods to adapt and predict comfortable human-robot proxemics. Proposing a network architecture, we experiment with three different layer configurations, obtaining three different end-to-end trainable models. Using these, we compare their predictive performances on data obtained during a human-robot interaction study. We find that our long short-term memory based model outperforms a gated recurrent unit based model and a feed-forward model. Further, we demonstrate how the created model can be used to create customized comfort zones that can help create a personalized experience for individual users.

Understanding emotional states is a challenging task which frequently leads to misinterpretation even in human observers. While the perception of emotions has been studied extensively in human psychology, little is known about what factors influence the human perception of emotions in robots and virtual characters. In this paper, we build on the Brunswik lens model to investigate the influence of (a) the agent's embodiment using a 2D virtual character, a 3D blended embodiment, a recording of the 3D platform and a recording of a human, as well as (b) the level of human-likeness on people's ability to interpret emotional facial expressions in an agent. In addition, we measure social traits of the human observers and analyze how they correlate to the success in recognizing emotional expressions. We find that interpersonal differences play a minor role in the perception of emotional states. However, both embodiment and human-likeness as well as related perceptual dimensions such as perceived social presence and uncanniness have an effect on the attribution of emotional states.

Mimicry plays an important role in social interaction. In human communication, it is used to establish rapport and bonding both with other humans, as well as robots and virtual characters. However, little is known about the underlying factors that elicit mimicry in humans when interacting with a robot. In this work, we study the influence of embodiment on participants' ability to mimic a social character. Participants were asked to intentionally mimic the laughing behavior of the Furhat mixed embodied robotic head and a 2D virtual version of the same character. To explore the effect of embodiment, we present two novel approaches to automatically assess people's ability to mimic based solely on videos of their facial expressions. In contrast to participants' self-assessment, the analysis of video recordings suggests a better ability to mimic when people interact with the 2D embodiment.

We present a study investigating the expressiveness of two different types of robots in a tutoring task. The robots used were i) the EMYS robot, with facial expression capabilities, and ii) the NAO robot, without facial expressions but able to perform expressive gestures. Preliminary results show that the NAO robot was perceived to be more friendly, pleasant and empathic than the EMYS robot as a tutor in a learning environment.

In this video submission, we describe a scenario developed in the EMOTE project. The overall goal of the EMOTE project is to develop an empathic robot tutor for 11-13 year old school students in an educational setting. The pedagogical domain here is to assist students in learning and testing their map-reading skills typically learned as part of the geography curriculum in schools. We show this scenario with a NAO robot interacting with the students whilst performing map-reading tasks on a touch-screen device in this video.

In this paper we propose a methodology for the creation of social interaction strategies for human-robot interaction based on restricted-perception Wizard-of-Oz studies (WoZ). This novel experimental technique involves restricting the wizard's perceptions over the environment and the behaviors it controls according to the robot's inherent perceptual and acting limitations. Within our methodology, the robot's design lifecycle is divided into three consecutive phases, namely data collection, where we perform interaction studies to extract expert knowledge and interaction data; strategy extraction, where a hybrid strategy controller for the robot is learned based on the gathered data; strategy refinement, where the controller is iteratively evaluated and adjusted. We developed a fully-autonomous robotic tutor based on the proposed approach in the context of a collaborative learning scenario. The results of the evaluation study show that, by performing restricted-perception WoZ studies, our robots are able to engage in very natural and socially-aware interactions.

The “social learning and multimodal interaction for designing artificial agents” workshop aims at presenting scientific and philosophical advances related to social learning and multimodal interaction for enhancing the design of artificial agents. Papers presented in the workshop include studies on human behavior modeling, on social robotics and on virtual agents. Our two invited speakers, Prof. Catherine Pelachaud and Prof. Louis-Philippe Morency will enrich and open the door to further discussion by bringing their widely acknowledged expertise in the field.

This paper investigates the influence of multimodal incongruent gender cues on the perception of a robot's uncanniness and gender in children. The back-projected robot head Furhat was equipped with a female and male face texture and voice synthesizer and the voice and facial cues were tested in congruent and incongruent combinations. 106 children between the age of 8 and 13 participated in the study. Results show that multimodal incongruent cues do not trigger the feeling of uncanniness in children. These results are significant as they support other recent research showing that the perception of uncanniness cannot be triggered by a categorical ambiguity in the robot. In addition, we found that children rely on auditory cues much stronger than on the facial cues when assigning a gender to the robot if presented with incongruent cues. These findings have implications for the robot design, as it seems possible to change the gender of a robot by only changing its voice without creating a feeling of uncanniness in a child.

Engagement in task orientated social robotics is a complex phenomenon, consisting of both task and social elements. Previous work in this area tends to focus on these aspects in isolation without consideration for the positive or negative effects one might cause the other. We explore both, in an attempt to understand how engagement with the task might effect the social relationship with the robot, and vice versa. In this paper, we describe the analysis of participant self-report data collected during an exploratory pilot study used to evaluate users' “perception of engagement”. We discuss how the results of our analysis suggest that ultimately, it was the users' own perception of the robots' characteristics such as friendliness, helpfulness and attentiveness which led to sustained engagement with both the task and robot.

In our work we explore the development of a computational model capable of automatically detecting engagement in social human-robot interactions from real-time sensory and contextual input. However, to train the model we need to establish ground truths of engagement from a large corpus of data collected from a study involving task and social-task engagement. Here, we intend to advance the current state-of-the-art by reducing the need for unreliable post-experiment questionnaires and costly time-consuming annotation with the novel introduction of implicit probes. A non-intrusive, pervasive and embedded method of collecting informative data at different stages of an interaction.

In this paper, we describe the results of an interview study conducted across several European countries on teachers' views on the use of empathic robotic tutors in the classroom. The main goals of the study were to elicit teachers' thoughts on the integration of the robotic tutors in the daily school practice, understanding the main roles that these robots could play and gather teachers' main concerns about this type of technology. Teachers' concerns were much related to the fairness of access to the technology, robustness of the robot in students' hands and disruption of other classroom activities. They saw a role for the tutor in acting as an engaging tool for all, preferably in groups, and gathering information about students' learning progress without taking over the teachers' responsibility for the actual assessment. The implications of these results are discussed in relation to teacher acceptance of ubiquitous technologies in general and robots in particular.

Social perception abilities are among the most important skills necessary for robots to engage humans in natural forms of interaction. Affect-sensitive robots are more likely to be able to establish and maintain believable interactions over extended periods of time. Nevertheless, the integration of affect recognition frameworks in real-time human-robot interaction scenarios is still underexplored. In this article, we propose and evaluate a context-sensitive affect recognition framework for a robotic game companion for children. The robot can automatically detect affective states experienced by children in an interactive chess game scenario. The affect recognition framework is based on the automatic extraction of task features and social interaction-based features. Vision-based indicators of the children’s nonverbal behaviour are merged with contextual features related to the game and the interaction and given as input to support vector machines to create a context-sensitive multimodal system for affect recognition. The affect recognition framework is fully integrated in an architecture for adaptive human-robot interaction. Experimental evaluation showed that children’s affect can be successfully predicted using a combination of behavioural and contextual data related to the game and the interaction with the robot. It was found that contextual data alone can be used to successfully predict a subset of affective dimensions, such as interest toward the robot. Experiments also showed that engagement with the robot can be predicted using information about the user’s valence, interest and anticipatory behaviour. These results provide evidence that social engagement can be modelled as a state consisting of affect and attention components in the context of the interaction.

This workshop covers real-time computational techniques for the recognition and interpretation of human affective and social behaviour, and techniques for synthesis of believable social behaviour supporting real-time adaptive human-agent and human-robot interaction in real-world environments.

The focus of this project is to design, develop and evaluate a new computational model for automatically detecting change in task engagement. This work will be applied to robotic tutors to enhance and support the learning experience, enabling timely pedagogical and empathic intervention. This work is intended to forward the current state of the art by 1) exploring how to automatically detect engagement with a learning task, 2) designing and developing new approaches to machine learning for adaptive platform-independent modelling and 3) evaluation of its effectiveness for building and maintaining learner engagement across different tutor embodiments, for example a physical and virtual embodiment.

In this paper we discuss the initial steps of the design of a computational model capable of automatically detecting user engagement with an artificial robotic tutor. Generally, our model will be used with a robotic tutor to support and enhance the experience of the learner by regulating pedagogical and empathetic interventions in a timely manner. We describe the requirements of the learner model in order to understand the state of the learner and facilitate the learning progress. Additionally, we propose the initial steps of the design of a suitable scenario for the learning task activity to allow the model to be tested on actual class material from UK curriculum based on teachers' feedback. Finally, we discuss an initial plan to build the model with real user data from a pilot study based on Wizard-of-Oz (WoZ) conducted in a real classroom environment.

This paper addresses the problem of feature fusion between smile, as a visual feature, and text, as a transcription result. The influence of smile over semantic data has been considered before, without investigating multiple approaches for the fusion. This problem is multi-modal, which makes it more difficult. The goal of this article is to investigate how this fusion could increase the current interactivity of a dialogue system by boosting the automatic detection rate of the sentiments expressed by a human user. There are two original propositions in our approach. The first lies in the use of a segmented detection for text data, rather than predicting a single label for every document (video). Second, this paper studies the importance of several features in the process of multi-modal fusion. Our approach uses basic features, such as NGrams, Smile Presence or Valence to find the best fusion approach. Moreover, we test a two level classification approach, using a SVM.

The idea of autonomous social robots capable of assisting us in our daily lives is becoming more real every day. However, there are still many open issues regarding the social capabilities that those robots should have in order to make daily interactions with humans more natural. For example, the role of affective interactions is still unclear. This paper presents an ethnographic study conducted in an elementary school where 40 children interacted with a social robot capable of recognising and responding empathically to some of the children's affective states. The findings suggest that the robot's empathic behaviour affected positively how children perceived the robot. However, the empathic behaviours should be selected carefully, under the risk of having the opposite effect. The target application scenario and the particular preferences of children seem to influence the degree of empathy that social robots should be endowed with.

Affect-sensitive systems such as social robots and virtual agents are increasingly being investigated in real-world settings. In order to work effectively in natural environments, these systems require the ability to infer the affective and mental states of humans and to provide appropriate timely output that helps to sustain long-term interactions. This special issue, which appears in two parts, includes articles on the design of socio-emotional behaviors and expressions in robots and virtual agents and on computational approaches for the automatic recognition of social signals and affective states.

Successful human interaction commonly involves prototypical exchanges where interactors are engaged, synchronized, and harmonious in their behaviors. The copying of aspects of the other's behavior, at different levels, seems central to establishing and maintaining such empathic connections. Yet, many questions remain unanswered, particularly how it is possible to reflect the same affective content back to the other when the actual motion itself is not exactly the same as theirs. This paper presents a perceptual study in which emotional gestures conducted by an actor were mapped onto synthesized versions generated by an embodied virtual agent. Copying is at the expressive level, where qualities such as the fluidity or expansiveness of gestures are considered, rather than exact low-level motion matching. Participants were later asked to rate the emotional content of video recordings of both the original and the synthesized gestures. A statistical analysis shows that, in most cases, participants associated the emotional content of the agent's gestures with that intended to be expressed by the original actor. The results suggest that a combination of the type of movement performed and its quality is important for successfully communicating emotions.

The design of an affect recognition system for socially perceptive robots relies on representative data: human-robot interaction in naturalistic settings requires an affect recognition system to be trained and validated with contextualised affective expressions, that is, expressions that emerge in the same interaction scenario of the target application. In this paper we propose an initial computational model to automatically analyse human postures and body motion to detect engagement of children playing chess with an iCat robot that acts as a game companion. Our approach is based on vision-based automatic extraction of expressive postural features from videos capturing the behaviour of the children from a lateral view. An initial evaluation, conducted by training several recognition models with contextualised affective postural expressions, suggests that patterns of postural behaviour can be used to accurately predict the engagement of the children with the robot, thus making our approach suitable for integration into an affect recognition system for a game companion in a real world scenario.

This paper gives a brief overview of the challenges for multi-model perception and generation applied to robot companions located in human social environments. It reviews the current position in both perception and generation and the immediate technical challenges and goes on to consider the extra issues raised by embodiment and social context. Finally, it briefly discusses the impact of systems that must function continually over months rather than just for a few hours.

The 3rd International Workshop on Affective Interaction in Natural Environments, AFFINE, follows a number of successful AFFINE workshops and events commencing in 2008.A key aim of AFFINE is the identification and investigation of significant open issues in real-time, affect-aware applications 'in the wild' and especially in embodied interaction, for example, with robots or virtual agents. AFFINE seeks to bring together researchers working on the real-time interpretation of user behaviour with those who are concerned with social robot and virtual agent interaction frameworks.

The Inter-ACT (INTEracting with Robots - Affect Context Task) corpus is an affective and contextually rich multimodal video corpus containing affective expressions of children playing chess with an iCat robot. It contains videos that capture the interaction from different perspectives and includes synchronised contextual information about the game and the behaviour displayed by the robot. The Inter-ACT corpus is mainly intended to be a comprehensive repository of naturalistic and contextualised, task-dependent data for the training and evaluation of an affect recognition system in an educational game scenario. The richness of contextual data that captures the whole human-robot interaction cycle, together with the fact that the corpus was collected in the same interaction scenario of the target application, make the Inter-ACT corpus unique in its genre.

Affect sensitivity is of the utmost importance for a robot companion to be able to display socially intelligent behaviour, a key requirement for sustaining long-term interactions with humans. This paper explores a naturalistic scenario in which children play chess with the iCat, a robot companion. A person-independent, Bayesian approach to detect the user's engagement with the iCat robot is presented. Our framework models both causes and effects of engagement: features related to the user's non-verbal behaviour, the task and the companion's affective reactions are identified to predict the children's level of engagement. An experiment was carried out to train and validate our model. Results show that our approach based on multimodal integration of task and social interaction-based features outperforms those based solely on non-verbal behaviour or contextual information (94.79 % vs. 93.75 % and 78.13 %).