Current approaches in education, while aiming to be universally effective, often struggle to fully adapt to the unique needs and communication styles of individual children; this disparity can limit the children’s engagement and hinder their learning progress. Similarly, parents or guardians, despite their good intentions, may also be unable to provide consistent and personalized support to each child. In this work, we investigate the use of conversational systems for socially assistive robots (SARs) as a potential solution to this problem, as such systems have the potential to allow children to interact and learn at their own pace, in a way that aligns with their communication preferences. To ensure that the robot’s language is suitable for children, we present a system that leverages a combination of natural language processing (NLP) techniques, including dialog management, child-friendly language generation, and context-aware response adaptation; to achieve this, our system combines Rasa for dialog management, GPT-3.5 for language generation, and textstat for language complexity evaluation. We evaluate the suitability of the generated language for a young audience through two user studies with adult participants, one in which the conversational system was embodied in a robot and involved direct interaction between a human and a robot, and another where participants evaluated conversational transcripts from the first study. Our results suggest that the system has the potential to maintain engaging and safe conversations, and adapt its language to individual needs.

Robot behaviour models in socially assistive robotics are typically trained using high-level features, such as a user’s engagement, such that inaccuracies in the feature extraction can have a significant effect on a robot’s subsequent performance. In this paper, we study whether a behaviour model can be meaningfully represented using an end-to-end approach, where multimodal input, concretely visual data and activity information, is directly processed by a neural network. This paper concretely analyses the different building blocks of such a model, such that the aim is to identify a suitable architecture that can meaningfully combine the different modalities for guiding a robot’s behaviour. We conduct the analysis in the context of a sequence learning game, such that we compare different vision-only models that are then combined with an activity processing network into a joint multimodal model. The results of our evaluation on a dedicated dataset from the sequence learning game demonstrate that a multimodal end-to-end behaviour model has potential for assistive robotics — we report an F1 score of around 0.88 across different dataset-based test scenarios — but the real-life transferability strongly depends on whether the data is diverse enough for capturing meaningful variations in real-world scenarios, such as users being at different distances from a robot.

In therapeutic scenarios, robots are sometimes used for imitation activities in which the robot demonstrates a motion and the individual under therapy needs to repeat it. To allow incorporating new types of motions in such activities, the robot should have an ability to learn motions by observing demonstrations from a human, such as a therapist. In this paper, we investigate an approach for acquiring motions from skeleton observations of a human, which are collected by a robot-centric RGB-D camera. The learning process from human body gestures to robot movements is done by mapping the joint angle positions to the robot’s body, such that self-collisions of the end effector are prevented by re-estimating the angles in a safe angular position. We performed both a quantitative and a qualitative evaluation of the method, namely we (i) quantitatively evaluated the motion reproduction error of the procedure by performing a study with QTrobot in which the robot acquired different upper-body dance moves from multiple participants, and (ii) performed a qualitative user study to evaluate the robot’s perceived reproduction. The quantitative evaluation demonstrates the method’s overall feasibility, although the reproduction quality is affected by noise in the skeleton observations, while the qualitative evaluation suggests generally high satisfaction with the robot’s motion, except for motions that are likely to lead to self-collisions and which were reproduced less accurately.

When faced with an execution failure, an intelligent robot should be able to identify the likely reasons for the failure and adapt its execution policy accordingly. This paper addresses the question of how to utilise knowledge about the execution process, expressed in terms of learned constraints, in order to direct the diagnosis and experience acquisition process. In particular, we present two methods for creating a synergy between failure diagnosis and execution model learning. We first propose a method for diagnosing execution failures of parameterised action execution models, which searches for action parameters that violate a learned precondition model. We then develop a strategy that uses the results of the diagnosis process for generating synthetic data that are more likely to lead to successful execution, thereby increasing the set of available experiences to learn from. The diagnosis and experience correction methods are evaluated for the problem of handle grasping, such that we experimentally demonstrate the effectiveness of the diagnosis algorithm and show that corrected failed experiences can contribute towards improving the execution success of a robot.

When an autonomous robot learns how to execute actions, it is of interest to know if and when the execution policy can be generalised to variations of the learning scenarios. This can inform the robot about the necessity of additional learning, as using incomplete or unsuitable policies can lead to execution failures. Generalisation is particularly relevant when a robot has to deal with a large variety of objects and in different contexts. In this paper, we propose and analyse a strategy for generalising parameterised execution models of manipulation actions over different objects based on an object ontology. In particular, a robot transfers a known execution model to objects of related classes according to the ontology, but only if there is no other evidence that the model may be unsuitable. This allows using ontological knowledge as prior information that is then refined by the robot’s own experiences. We verify our algorithm for two actions - grasping and stowing everyday objects - such that we show that the robot can deduce cases in which an existing policy can generalise to other objects and when additional execution knowledge has to be acquired.

An essential measure of autonomy in assistive service robots is adaptivity to the various contexts of human-oriented tasks, which are subject to subtle variations in task parameters that determine optimal behaviour. In this work, we propose an apprenticeship learning approach to achieving context-aware action generalization on the task of robot-to-human object hand-over. The procedure combines learning from demonstration and reinforcement learning: a robot first imitates a demonstrator's execution of the task and then learns contextualized variants of the demonstrated action through experience. We use dynamic movement primitives as compact motion representations, and a model-based C-REPS algorithm for learning policies that can specify hand-over position, conditioned on context variables. Policies are learned using simulated task executions, before transferring them to the robot and evaluating emergent behaviours. We additionally conduct a user study involving participants assuming different postures and receiving an object from a robot, which executes hand-overs by either imitating a demonstrated motion, or adapting its motion to hand-over positions suggested by the learned policy. The results confirm the hypothesized improvements in the robot's perceived behaviour when it is context-aware and adaptive, and provide useful insights that can inform future developments.

For robots acting in human-centered environments, the ability to improve based on experience is essential for reliable and adaptive operation; however, particularly in the context of robot failure analysis, experience-based improvement is practically useful only if robots are also able to reason about and explain the decisions they make during execution. In this paper, we describe and analyse a representation of execution-specific knowledge that combines (i) a relational model in the form of qualitative attributes that describe the conditions under which actions can be executed successfully and (ii) a continuous model in the form of a Gaussian process that can be used for generating parameters for action execution, but also for evaluating the expected execution success given a particular action parameterisation. The proposed representation is based on prior, modelled knowledge about actions and is combined with a learning process that is supervised by a teacher. We analyse the benefits of this representation in the context of two actions - grasping handles and pulling an object on a table -such that the experiments demonstrate that the joint relational-continuous model allows a robot to improve its execution based on experience, while reducing the severity of failures experienced during execution.

While executing actions, service robots may experience external faults because of insufficient knowledge about the actions' preconditions. The possibility of encountering such faults can be minimised if symbolic and geometric precondition models are combined into a representation that specifies how and where actions should be executed. This work investigates the problem of learning such action execution models and the manner in which those models can be generalised. In particular, we develop a template-based representation of execution models, which we call δ models, and describe how symbolic template representations and geometric success probability distributions can be combined for generalising the templates beyond the problem instances on which they are created. Our experimental analysis, which is performed with two physical robot platforms, shows that δ models can describe execution-specific knowledge reliably, thus serving as a viable model for avoiding the occurrence of external faults.