Learning from Demonstrations (LfD) allows robots to learn skills from human users, but its effectiveness can suffer due to sub-optimal teaching, especially from untrained demonstrators. Active LfD aims to improve this by letting robots actively request demonstrations to enhance learning. However, this may lead to frequent context switches between various task situations, increasing the human cognitive load and introducing errors to demonstrations. Moreover, few prior studies in active LfD have examined how these active query strategies may impact human teaching in aspects beyond user experience, which can be crucial for developing algorithms that benefit both robot learning and human teaching. To tackle these challenges, we propose an active LfD method that optimizes the query sequence of online human demonstrations via Curriculum Learning (CL), where demonstrators are guided to provide demonstrations in situations of gradually increasing difficulty. We evaluate our method across four simulated robotic tasks with sparse rewards and conduct a user study$(N=26)$to investigate the influence of active LfD methods on human teaching regarding teaching performance, post-guidance teaching adaptivity, and teaching transferability. Our results show that our method significantly improves learning performance compared to three other LfD baselines in terms of the final success rate of the converged policy and sample efficiency. Additionally, results from our user study indicate that our method significantly reduces the time required from human demonstrators and decreases failed demonstration attempts. It also enhances post-guidance human teaching in both seen and unseen scenarios compared to another active LfD baseline, indicating enhanced teaching performance, greater postguidance teaching adaptivity, and better teaching transferability achieved by our method.

Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.

The increasing integration of robots into human environments necessitates efficient learning systems capable of adapting to complex scenarios while co-existing with humans. Traditional reinforcement learning (RL) is one of the most popular option, but often struggles with inefficiencies, such as sparse rewards and prolonged training. Learning from Demonstration (LfD), which leverages human expertise, offers a promising alternative. However, human teaching strategies and robot learning processes are inherently intertwined in LfD. Ineffective human teaching can diminish robot learning. To effectively provide demonstrations, human teachers require an understanding of the robot’s internal processes and needs without being overwhelmed. We address this by visually showing the robot’s deviation from expectation, a metric based on Temporal Difference (TD) error, which represents discrepancies between predicted and actual outcomes. We conducted a user study (n=12) comparing two conditions: one in which deviations from expectation were visually indicated, and one in which these deviations were not shown. Results indicate that visualising deviations shifts human teaching behavior from result oriented strategy (providing demonstrations in the areas where the robot fails) to an expectation oriented strategy (focusing on demonstrations where robot’s deviation from expectation is high). We conducted a follow-up simulation study to investigate how these two teaching strategies may influence robot learning, showing that diverse and widespread demonstrations have a significant effect on robot learning performance. We conclude our work with actionable guidelines for designing human-robot interactions that better align human teaching behaviors with robot learning requirements.

Reinforcement Learning (RL) has achieved great success in sequential decision-making problems but often requires extensive agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) incorporate external expert demonstrations to aid agent exploration during the learning process. However, these demonstrations, typically collected from human users, are costly and thus often limited in quantity. Therefore, how to select the optimal set of human demonstrations that most effectively aids learning becomes a critical concern. This paper introduces EARLY (Episodic Active Learning from demonstration querY), an algorithm designed to enable a learning agent to generate optimized queries for expert demonstrations in a trajectory-based feature space. EARLY employs a trajectory-level estimate of uncertainty in the agent’s current policy to determine the optimal timing and content for feature-based queries. By querying episodic demonstrations instead of isolated state-action pairs, EARLY enhances the human teaching experience and achieves better learning performance. We validate the effectiveness of our method across three simulated navigation tasks of increasing difficulty. Results indicate that our method achieves expert-level performance in all three tasks, converging over 50% faster than other four baseline methods when demonstrations are generated by simulated oracle policies. A follow-up pilot user study (N = 18) further supports that our method maintains significantly better convergence with human expert demonstrators, while also providing a better user experience in terms of perceived task load and requiring significantly less human time.

Learning from Demonstrations (LfD) transfers skills from human teachers to robots. However, data imbalance in demonstrations can bias policies towards majority situations. Previous work attempted to solve this problem after data collection, but few efforts were made to maintain a balanced distribution from the phase of data acquisition. Our method accounts for the influence of robots on human teachers and enables robots to actively guide interaction to approximate demonstration distributions to target distributions. Simulated and real-world experiments validated the method’s efficacy in shaping demonstration distribution into various target distributions and robustness to various levels of uncertainties. Also, our method significantly improved the generalization ability of robot learning when LfD policies were trained with data collected by our method compared to natural data collection.

Human-centered interactive robot tasks (e.g., social greetings and cooperative dressing) are a type of task where humans are involved in task dynamics and performance evaluation. Such tasks require spatial and temporal coordination between agents in real-time, tackling physical limitations from constrained robot bodies, and connecting human user experience with concrete learning objectives to inform algorithm design. To solve these challenges, imitation learning has become a popular approach where by a robot learns to perform a task by imitating how human experts do it (i.e., expert policies). However, previous works tend to isolate the algorithm design from the design of the whole learning pipeline, neglecting its connection with other modules inside the process (like data collection and user-centered subjective evaluation) from the view as a system. Going beyond traditional imitation learning, this work reexamines robot imitation learning in human-centered interactive tasks from the perspective of the whole learning pipeline, ranging from data collection to subjective evaluation. We present a process-oriented framework that consists of a guideline to collect diverse yet representative demonstrations and an interpreter to explain subjective user-centered performance with objective robot-related parameters. We illustrate the steps covered by the framework in a fist-bump greeting task as demonstrative deployment. Results show that our framework is able to identify representative human-centered features to instruct demonstration collection and validate influential robot-centered factors to interpret the gap in subjective performance between the expert policy and the imitator policy.