Robot learning from humans has been proposed and researched for several decades as a means to enable robots to learn new skills or adapt existing ones to new situations. Recent advances in AI, including learning approaches like reinforcement learning and architectures like transformers and foundation models, combined with access to massive datasets, have created attractive opportunities to apply those data-hungry techniques to this problem. We argue that the focus on massive amounts of pre-collected data, and the resulting learning paradigm, where humans demonstrate and robots learn in isolation, is overshadowing a specialized area of work we term Human-Interactive Robot Learning (HIRL). This paradigm, wherein robots and humans interact during the learning process , is at the intersection of multiple fields (AI, robotics, human–computer interaction, design and others) and holds unique promise. Using HIRL, robots can achieve greater sample efficiency (as humans can provide task knowledge through interaction), align with human preferences (as humans can guide the robot behavior toward their expectations), and explore more meaningfully and safely (as humans can utilize domain knowledge to guide learning and prevent catastrophic failures). This can result in robotic systems that can more quickly and easily adapt to new tasks in human environments. The objective of this article is to provide a broad and consistent overview of HIRL research and to guide researchers toward understanding the scope of HIRL, and current open or underexplored challenges related to four themes—namely, human, robot learning, interaction, and broader context. The article includes concrete use cases to illustrate the interaction between these challenges and inspire further research according to broad recommendations and a call for action for the growing HIRL community.

Learning from Demonstrations (LfD) allows robots to learn skills from human users, but its effectiveness can suffer due to sub-optimal teaching, especially from untrained demonstrators. Active LfD aims to improve this by letting robots actively request demonstrations to enhance learning. However, this may lead to frequent context switches between various task situations, increasing the human cognitive load and introducing errors to demonstrations. Moreover, few prior studies in active LfD have examined how these active query strategies may impact human teaching in aspects beyond user experience, which can be crucial for developing algorithms that benefit both robot learning and human teaching. To tackle these challenges, we propose an active LfD method that optimizes the query sequence of online human demonstrations via Curriculum Learning (CL), where demonstrators are guided to provide demonstrations in situations of gradually increasing difficulty. We evaluate our method across four simulated robotic tasks with sparse rewards and conduct a user study$(N=26)$to investigate the influence of active LfD methods on human teaching regarding teaching performance, post-guidance teaching adaptivity, and teaching transferability. Our results show that our method significantly improves learning performance compared to three other LfD baselines in terms of the final success rate of the converged policy and sample efficiency. Additionally, results from our user study indicate that our method significantly reduces the time required from human demonstrators and decreases failed demonstration attempts. It also enhances post-guidance human teaching in both seen and unseen scenarios compared to another active LfD baseline, indicating enhanced teaching performance, greater postguidance teaching adaptivity, and better teaching transferability achieved by our method.

Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.

As robots will be deployed around non-expert users in unstructured environments, they are bound to fail or behave suboptimally in some cases. Through interactive learning approaches, robots can leverage interaction with non-expert users to refine existing skills or even learn new tasks. If we will (at least partially) rely on these users to occasionally teach robots, we need to make sure the teaching tasks is seen as enjoyable rather than a burden. To this end, this paper presents a preliminary system design and evaluation for a multi-modal teaching system for non-expert users to teach simple navigation tasks to mobile robots. Our design, using human-dog interaction as a metaphor, focuses on creating engaging and embodied interactions that foster a sense of participation. Our preliminary results (N=20) suggest that participants had a positive perception of the learning system, and showed a clear preference for demonstrations over feedback as teaching signal. Participants with more affinity and experience with technology found the system more intuitive and engaging, and provided more accurate demonstrations, underscoring the importance of considering the users’ background and expertise when designing learning systems. This work paves the way towards designing usable and engaging teaching interfaces for robots that reduce barriers for non-expert users and foster a sense of teaching as an act of "care".

Human-interactive robot learning allows a robot to learn tasks more effectively with the help of humans in the role of teacher. While there is a large body of work on algorithms that leverage human input for better robot learning, there has been little attention to understanding how humans teach robots. In this paper, we provide preliminary results on how users strategize the use of demonstrations and evaluative feedback under a budget, and how these choices are influenced by demographic variables such as gender. We implemented a learning algorithm that allows a simulated robot arm to learn three reaching tasks with the help of a human. We collected interaction data for a total of 58 participants, which shows that participants demonstrate a tendency to provide evaluative feedback earlier in their interactions compared to demonstrations, and that gender may have an influence on teaching strategy. This preliminary analysis lays the foundation for future research aimed at developing tuneable computational models of different human teachers.

The increasing integration of robots into human environments necessitates efficient learning systems capable of adapting to complex scenarios while co-existing with humans. Traditional reinforcement learning (RL) is one of the most popular option, but often struggles with inefficiencies, such as sparse rewards and prolonged training. Learning from Demonstration (LfD), which leverages human expertise, offers a promising alternative. However, human teaching strategies and robot learning processes are inherently intertwined in LfD. Ineffective human teaching can diminish robot learning. To effectively provide demonstrations, human teachers require an understanding of the robot’s internal processes and needs without being overwhelmed. We address this by visually showing the robot’s deviation from expectation, a metric based on Temporal Difference (TD) error, which represents discrepancies between predicted and actual outcomes. We conducted a user study (n=12) comparing two conditions: one in which deviations from expectation were visually indicated, and one in which these deviations were not shown. Results indicate that visualising deviations shifts human teaching behavior from result oriented strategy (providing demonstrations in the areas where the robot fails) to an expectation oriented strategy (focusing on demonstrations where robot’s deviation from expectation is high). We conducted a follow-up simulation study to investigate how these two teaching strategies may influence robot learning, showing that diverse and widespread demonstrations have a significant effect on robot learning performance. We conclude our work with actionable guidelines for designing human-robot interactions that better align human teaching behaviors with robot learning requirements.

Reinforcement Learning (RL) has achieved great success in sequential decision-making problems but often requires extensive agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) incorporate external expert demonstrations to aid agent exploration during the learning process. However, these demonstrations, typically collected from human users, are costly and thus often limited in quantity. Therefore, how to select the optimal set of human demonstrations that most effectively aids learning becomes a critical concern. This paper introduces EARLY (Episodic Active Learning from demonstration querY), an algorithm designed to enable a learning agent to generate optimized queries for expert demonstrations in a trajectory-based feature space. EARLY employs a trajectory-level estimate of uncertainty in the agent’s current policy to determine the optimal timing and content for feature-based queries. By querying episodic demonstrations instead of isolated state-action pairs, EARLY enhances the human teaching experience and achieves better learning performance. We validate the effectiveness of our method across three simulated navigation tasks of increasing difficulty. Results indicate that our method achieves expert-level performance in all three tasks, converging over 50% faster than other four baseline methods when demonstrations are generated by simulated oracle policies. A follow-up pilot user study (N = 18) further supports that our method maintains significantly better convergence with human expert demonstrators, while also providing a better user experience in terms of perceived task load and requiring significantly less human time.

Agent learning from human interaction often relies on explicit signals, but implicit social cues, such as prosody in speech, could provide valuable information for more effective learning. This paper advocates for the integration of prosody as a teaching signal to enhance agent learning from human teachers. Through two exploratory studies—one examining voice feedback in an interactive reinforcement learning setup and the other analyzing restricted audio from human demonstrations in three Atari games—we demonstrate that prosody carries significant information about task dynamics. Our findings suggest that prosodic features, when coupled with explicit feedback, can enhance reinforcement learning outcomes. Moreover, we propose guidelines for prosody-sensitive algorithm design and discuss insights into teaching behavior. Our work underscores the potential of leveraging prosody as an implicit signal for more efficient agent learning, thus advancing human-agent interaction paradigms.

Recent models for Visual Speech Recognition (VSR) have shown remarkable progress over the last few years. They have however been applied mainly to datasets such as Lip Reading Sentences 3 (LRS3), LRS2 or Lombard GRID, but not yet on social robots. As social robots struggle to recognize speech in more challenging acoustic and crowded environments, we believe such models are promising tools for real-time interaction with users. This paper presents a feasibility study focusing on integration of speech recognition (SR) using mixed modalities - audio, visual (lip-reading) and audio-visual - in social robots. To this end, this paper contributes a pipeline to detect an active speaker based on lip movement, post-processing of audio and video footage and inferencing it with the state-of-the-art Auto-AVSR model. In a user study (N = 26), we evaluated the feasibility of audio, visual and mixed modality speech recognition on a Pepper robot. We demonstrate the feasibility of using singular and mixed modalities with speech-to-text inference in natural interaction. The results show that it is feasible to deploy such models on social robots in a controlled, noiseless and non-interactive environment. Additionally, the results revealed that informing participants to emphasize their lip movements significantly improved text-to-speech inference results. Our work provides initial insights into the benefits and challenges of using VSR, ASR and AVSR for HRI.

Large language models, in particular generative pre-trained transformers (GPTs), show impressive results on a wide variety of language-related tasks. In this paper, we explore ChatGPT’s zero-shot ability to perform affective computing tasks using prompting alone. We show that ChatGPT a) performs meaningful sentiment analysis in the Valence, Arousal and Dominance dimensions, b) has meaningful emotion representations in terms of emotion categories and these affective dimensions, and c) can perform basic appraisal-based emotion elicitation of situations based on a prompt-based computational implementation of the OCC appraisal model. These findings are highly relevant: First, they show that the ability to solve complex affect processing tasks emerges from language-based token prediction trained on extensive data sets. Second, they show the potential of large language models for simulating, processing and analyzing human emotions, which has important implications for various applications such as sentiment analysis, socially interactive agents, and social robotics.

This paper presents a novel device that can be used to perform kinesthetic corrective feedback for robotic systems. KRIS (Kinesthetic Robotic Interaction System) is a device that can be mounted on the end-effector of an articulated robot. From here it can be manipulated by a human to give corrective feedback to the robot system during execution and in an intuitive way. The device can provide feedback in six degrees of freedom while giving passive haptic feedback to the user about both the position, rotation, and movement of the robot. We evaluated KRIS in a user study with respect to a baseline based on keyboard feedback in the areas of usability, intuitiveness, accuracy of corrections, and user task load. KRIS outperformed our baseline on the first three metrics and performed similar on task load. We believe that KRIS can enable a wide variety of robots to be taught interactively by non-expert humans in diverse collaborative settings.

Learning from Demonstrations (LfD) transfers skills from human teachers to robots. However, data imbalance in demonstrations can bias policies towards majority situations. Previous work attempted to solve this problem after data collection, but few efforts were made to maintain a balanced distribution from the phase of data acquisition. Our method accounts for the influence of robots on human teachers and enables robots to actively guide interaction to approximate demonstration distributions to target distributions. Simulated and real-world experiments validated the method’s efficacy in shaping demonstration distribution into various target distributions and robustness to various levels of uncertainties. Also, our method significantly improved the generalization ability of robot learning when LfD policies were trained with data collected by our method compared to natural data collection.

Human-centered interactive robot tasks (e.g., social greetings and cooperative dressing) are a type of task where humans are involved in task dynamics and performance evaluation. Such tasks require spatial and temporal coordination between agents in real-time, tackling physical limitations from constrained robot bodies, and connecting human user experience with concrete learning objectives to inform algorithm design. To solve these challenges, imitation learning has become a popular approach where by a robot learns to perform a task by imitating how human experts do it (i.e., expert policies). However, previous works tend to isolate the algorithm design from the design of the whole learning pipeline, neglecting its connection with other modules inside the process (like data collection and user-centered subjective evaluation) from the view as a system. Going beyond traditional imitation learning, this work reexamines robot imitation learning in human-centered interactive tasks from the perspective of the whole learning pipeline, ranging from data collection to subjective evaluation. We present a process-oriented framework that consists of a guideline to collect diverse yet representative demonstrations and an interpreter to explain subjective user-centered performance with objective robot-related parameters. We illustrate the steps covered by the framework in a fist-bump greeting task as demonstrative deployment. Results show that our framework is able to identify representative human-centered features to instruct demonstration collection and validate influential robot-centered factors to interpret the gap in subjective performance between the expert policy and the imitator policy.

To develop and sustain rich social interactions between humans and robots, previous research has mostly looked at task-oriented performance metrics or the ability for a robot to adequately express messages, emotions, or intents. In contrast, our research starts from the premise that movement, as a nonverbal modality of social interaction, can cover other essential aspects of social interaction that do not have to do with the expression of messages, inner states, or drives but that nonetheless contribute to improving the quality of interaction. These aspects have to do with interaction dynamics and highly depend on appropriate action choice. Drawing inspiration from rule-based improvisation, this paper seeks to show that there exists implicit expert knowledge that can be used to inform these movement action choices, contributing to rich, playful, and non goal-oriented interactions between humans and robots. We present an experimental study conducted at a performing arts festival, in which participants interacted with a robot in three simple rule-based movement games, in two conditions: one where the robot was fully controlled by an improvisation expert (Improv Timing/Improv Action) and one where the timing of the actions was controlled by the expert but the robot’s action choices were drawn randomly (Improv Timing/Random Action). This was done in order to focus on action choice, beyond the timing of a response. Our results show that the Improv Timing/Improv Action condition not only performs better in terms of anthropomorphism and animacy, but also increases the interest of people in interacting with the robot for longer periods of time. These results serve as preliminary evidence of how improvisational knowledge in this context contributes to improving the quality of an interaction, and point at the value of further work in this field.

We present CycleDance, a dance style transfer system to transform an existing motion clip in one dance style to a motion clip in another dance style while attempting to preserve motion context of the dance. Our method extends an existing CycleGAN architecture for modeling audio sequences and integrates multimodal transformer encoders to account for music context. We adopt sequence length-based curriculum learning to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. We further introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. We perform an extensive ablation study as well as a human study including 30 participants with 5 or more years of dance experience. The results demonstrate that CycleDance generates realistic movements with the target style, significantly outperforming the baseline CycleGAN on naturalness, transfer strength, and content preservation.1

Abstract This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.

With robots poised to enter our daily environments, we conjecture that they will not only need to work for people, but also learn from them. An active area of investigation in the robotics, machine learning, and human-robot interaction communities is the design of teachable robotic agents that can learn interactively from human input. To refer to these research efforts, we use the umbrella term Human-Interactive Robot Learning (HIRL). While algorithmic solutions for robots learning from people have been investigated in a variety of ways, HIRL, as a fairly new research area, is still lacking: 1) a formal set of definitions to classify related but distinct research problems or solutions, 2) benchmark tasks, interactions, and metrics to evaluate the performance of HIRL algorithms and interactions, and 3) clear long-term research challenges to be addressed by different communities. The main goal of this workshop will be to consolidate relevant recent work falling under the HIRL umbrella into a coherent set of long, medium, and short-term research problems, and identify the most pressing future research goals in this area. As HIRL is a developing research area, this workshop is an opportunity to break the existing boundaries between relevant research communities by developing and sharing a diverse set of benchmark tasks and metrics for HIRL, inspired by other fields including neuroscience, biology, and ethics research.

A key challenge in human-robot interaction (HRI) design is to create and sustain engaging social interactions. This paper argues that improvisational techniques from the performing arts can address this challenge. Contrary to the ways in which improvisation is generally used in social robotics, we propose an understanding of improvisational techniques as based on rules that shape motion choices. We claim that such an approach, represented in what we name the “external” and “emergent” perspectives on improvisation, could benefit the way in which robot movement and behaviour is designed and deployed, increasing playful engagement and responsiveness. As an example of this type of improvisation, we discuss how American dancer and choreographer William Forsythe's Improvisation Technologies could be used in an HRI context. We also report on a preliminary experimentation using a Wizard-of-Oz exploratory prototyping system and a participatory design method with professional dancers geared towards the exploration of interactive movement possibilities with a Pepperrobot. Finally, we report on how this workshop offered valuable information about the applicability of these tools, as well as reflections on how it could help increase the level of engagement in the interaction.

In order to be successfully integrated into human-populated environments, mobile robots need to express relevant information about their state to the outside world. In particular, animated lights are a promising way to express hidden robot state information such that it is visible at a distance. In this work, we present an online study to evaluate the effect of robot communication through expressive lights on people's understanding of the robot's state and actions. In our study, we use the CoBot mobile service robot with our light interface, designed to express relevant robot information to humans. We evaluate three designed light animations on three corresponding scenarios for each, for a total of nine scenarios. Our results suggest that expressive lights can play a significant role in helping people accurately hypothesize about a mobile robot's state and actions from afar when minimal contextual clues are present. We conclude that lights could be generally used as an effective non-verbal communication modality for mobile robots in the absence of, or as a complement to, other modalities.