While Language Models have shown promising performance, they still struggle with limitations regarding reasoning and are very token-sensitive. In contrast, knowledge-based systems, such as ontologies, allow for provable logically valid reasoning and provide explicit justifications regarding newly inferred knowledge. However, those justifications can be hard to understand for non-expert users given their formal syntax and their length. We investigated if language models could be considered as reliable tools for verbalizing such explanations, thus increasing explainability over reasoning output. This paper presents a reference evaluation of a set of embeddable language models on a task of translation from rule-based ontology formatted inferences and justifications into natural language sentences. We show that the order of justifications significantly decreases performance, whereas adding the inference rule as additional context significantly improves performance, leading to more reliable results.

When collaborating with humans during a shared task, a robot must be able to estimate the shared goal and monitor the tasks completed by its partners to adapt its behavior. Our contribution is a lightweight, hierarchical task recognition system that enables the robot to estimate shared goals and monitor human tasks. This recognition system is integrated into a robotic architecture to take advantage of the semantic knowledge flow available and builds upon our previous work on action recognition. We demonstrate the mechanisms of our recognition system and how we improved it to handle missing information using a kitchen scenario. This also enables us to showcase its usability from the perspective of other agents, using Theory of Mind.

Coexisting with others and interacting in society implies sharing knowledge and attention about world objects, events, features, episodes, and even imagination or abstract ideas in time and space. Inspired by human phenomenological, cognitive and behavioral research, this work focuses on the study of joint attention (JA) for human-robot interaction (HRI), based on two main assumptions: a) the perception and representation of attention jointness constitute an isomorphic relation, and b) inspiration on dynamic neural fields (DNF) theory is a promising way to investigate contextual and non-linear spatio-temporal relations underlying attention and knowledge sharing in HRI. Taking into account the previous considerations, we propose a topology-based model for JA named TOP-JAM, which is able to represent and track in real-time JA states, from observations of behavioral data. More importantly, the model consists in a representation that can be directly understood by human beings, which conforms to robo-ethical principles in social robotics. This study evaluates computational properties of the model in simulation. Through a real experiment with the robot Pepper, the study shows that TOP-JAM is able to track JA in a triad interaction scenario.

In collaborative tasks, expectations for achieving shared goals arise at all hierarchical plan levels, including plans, tasks, subtasks, and actions. However, these expectations also generate uncertainties for individuals executing the joint plan. If left unresolved, these uncertainties can impede successful task completion. Uncertainties may relate to the agents’ motivation to initiate, continue, or complete their plan (motivational uncertainty), the best way to execute their shared plan (instrumental uncertainty), and their knowledge of other agents and the environment (common ground uncertainty). These expectations can be either normative or descriptive, but only normative expectations trigger reactions from agents to resolve the aforementioned types of uncertainties. Thus, this paper introduces a theoretical model that enables a robot to consider all agents’ expectations and take actions that reduce the uncertainties associated with their shared plan. By doing so, we aim to enhance the likelihood of success in joint plans between robots and humans. To demonstrate the effectiveness of our theoretical commitment model, we have implemented a proof of concept for a client service use case in a food shop.

Navigation of robots among humans is still an open problem, especially in confined locations (e.g. narrow corridors, doors). This article aims at finding how an anthropomorphic robot, like a PR2 robot with a height of 1.33 m, should behave when crossing a human in a narrow corridor in order to increase its usability. Two experiments studied how a combination of robot head behavior and navigation strategy can enhance robot legibility. Experiment 1 aimed to measure where a pedestrian looks when crossing another pedestrian, comparing the nature of the pedestrian: human or a robot. Based on the results of this experiment and the literature, we then designed a robot behavior exhibiting mutual manifestness by both modifying its trajectory to be more legible, and using its head to glance at the human. Experiment 2 evaluated this behavior in real situations of pedestrians crossing a robot. The visual behavior and user experience of pedestrians were assessed. The first experiment revealed that humans primarily look at the robot's head just before crossing. The second experiment showed that when crossing a human in a narrow corridor, both modifying the robot trajectory and glancing at the human is necessary to significantly increase the usability of the robot. We suggest using mutual manifestness is crucial for an anthropomorphic robot when crossing a human in a corridor. It should be conveyed both by altering the trajectory and by showing the robot awareness of the human presence through the robot head motion. Small changes in robot trajectory and manifesting robot perception of the human via a user identified robot head can avoid users' hesitation and feeling of threat.

The supervision component is the binder of a robotic architecture. Without it, there is no task, no interaction happening, it conducts the other components of the architecture towards the achievement of a goal, which means, in the context of a collaboration with a human, to bring changes in the physical environment and to update the human partner mental state. However, not so much work focus on this component in charge of the robot decision-making and control, whereas this is the robot puppeteer. Most often, either tasks are simply scripted, or the supervisor is built for a specific task. Thus, we propose JAHRVIS, a Joint Action-based Human-aware supeRVISor. It aims at being task-independent while implementing a set of key joint action and collaboration mechanisms. With this contribution, we intend to move the deployment of autonomous collaborative robots forward, accompanying this paper with our open-source code.

Being able to refer to an object, a person, or a place in a non-ambiguous manner is a need when one has to achieve collaborative activities with a partner. This is the so-called Referring Expression Generation (REG) problem. While widely used for Human-Robot Interaction, state of the art approaches restrict its use to the current environment. We propose a novel extension to the REG which takes full advantage of the Human-Robot shared knowledge about past actions as additional information to generate Referring Expressions. We show that our approach is usable with a domain-independent ontology as a knowledge base and that it can also use a semantic representation of past activity to generate RE. We illustrate our method through simulated situations and discuss its efficiency and pertinence.

Assessing robotic architecture for Human-Robot Interaction can be challenging due to the number of features a robot has to endow to perform an acceptable interaction. While everyday-inspired tasks are interesting as reflecting a realistic use of such robots, they often contain a lot of unknown and uncontrolled conditions and specific robot behavior can be hard to test. In this paper, we propose a new psychology-inspired task, gathering perspective-taking, planning, knowledge representation with theory of mind, manipulation, and communication. Along with a precise description of the task allowing its replication, we present a cognitive robot architecture able to perform it in its nominal cases. We finally suggest some challenges and evaluations for the Human-Robot Interaction research community, all derived from this easy-to-replicate task.

In Human-Robot Interaction (HRI), ensuring nonambiguous communication between the robot and the human is a key point for carrying out fluently a collaborative task. With this work, we propose a method which allows the robot to generate the optimal set of assertions that are necessary in order to produce an unambiguous reference. In this paper, we present a novel approach to the Referring Expression Generation (REG) problem and its integration into a robotic system. Our method is a domain-independent approach based on an ontology as a knowledge base. We show how this generation can be performed on an ontology which is not dedicated to this task. We then validate our method through simulated situations, compare it with state of the art approach and on a real robotic system.

When we perform a collaborative task with another human, we are able to tell, to a certain extent, how things are going and more precisely if things are going well or not. This knowledge allows us to adapt our behavior. Therefore, we think it is desirable to provide robots with means to measure in real-time the Quality of the Interaction with their human partners. To make this possible, we propose a model and a set of metrics targeting the evaluation of the QoI in collaborative tasks through the measure of the human engagement and the online task effectiveness. These model and metrics have been implemented and tested within the high-level controller of an entertainment robot deployed in a mall. The first results show significant differences in the computed QoI when in interaction with a fully compliant human, a confused human and a non-cooperative one.

Reasoning about spatial and geometric relations between objects in a tabletop human-robot interaction is a challenge due to the perception not being always consistent: objects placed on a table seem to be slightly in the air; they overlap; they disappear due to occlusions. Yet, interpreting and anchoring perceptual data in a physically consistent estimation of the scene is a crucial ability for humans, and thus robots in HRI context. In this paper we present a simulation-based physics reasoner integrated in a lightweight situation-assessment framework called Underworlds, that allows the robot to stabilize objects and build at run-time a consistent estimation of the scene, even for entirely hidden objects, while inferring the actions performed by its human partner.

In this paper we present Ontologenius, a semantic knowledge storage and reasoning framework for autonomous robots. More than a classic ontology software to query a knowledge base and a first-order internal logic as it can be done for web-semantics, we propose with Ontologenius features adapted to a robotic use including human-robot interaction. We introduce the ability to modify the knowledge base during execution, whether through dialogue or geometric reasoning, and keep these changes even after the robot is powered off. Since Ontologenius was developed to be used by a robot which interacts with humans, we have endowed the system with ability to perform attributes and properties generalization and with the possibility to model and estimate the semantic memory of a human partner and to implement theory of mind processes. This paper presents the architecture and the main features of Ontologenius as well as examples of its use in robotics applications.

We claim that the activity consisting in providing route directions can be best dealt with as a joint task involving the contribution not only of the robot as a direction provider but also of the human as listener. Moreover, we claim that in some cases, both the robot and the human should move to reach a different perspective of the environment which allows the explanations to be more efficient. As a first step toward implementing such a system, we propose the SVP (Shared Visual Perspective) planner which searches for the right placements both for the robot and the human to enable the visual perspective sharing needed for providing route direction and which makes the choice of the best landmark when several are available. The shared perspective is chosen taking into account not only the visibility of the landmarks, but the whole guiding task.

We introduce UNDERWORLDS, a novel lightweight framework for cascading spatio-temporal situation assessment in robotics. UNDERWORLDS allows programmers to represent the robot's environment as real-time distributed data structures, containing both scene graphs (for representation of 3D geometries) and timelines (for representation of temporal events). UNDERWORLDS supports cascading representations: the environment is viewed as a set of worlds that can each have different spatial and temporal granularities, and may inherit from each other. UNDERWORLDS also provides a set of high-level client libraries and tools to introspect and manipulate the environment models. This article presents the design and architecture of this open-source tool, and explores some applications, along with examples of use.

The domain of human-robot Joint Action is a growing field where roboticists, psychologists and philosophers start to collaborate in order to devise robot abilities that are as efficient and convenient for the human partner as possible. Besides studying Joint Action and developing algorithms and schemes to control the robot and manage the interaction, one of the current challenges is to come up with a method to properly evaluate the progresses made by the community. Several questionnaires have already been proposed to the community that deal with the evaluation of human-robot interaction. However, these studies mainly concern either specific basic behaviors during Joint Action or human-robot interactions without effective physical Joint Action. When it comes to high level decisions during physical human-robot Joint Action, there are fewer contributions to the topic, and also, the methods to evaluate them are even rarer. The aim of this paper is to propose a reusable questionnaire PeRDITA (Pertinence of Robot Decisions In joinT Action) allowing us to evaluate the pertinence of high level decision abilities of a robot during physical Joint Action with a human.

Human–Robot Interaction challenges Artificial Intelligence in many regards: dynamic, partially unknown environments that were not originally designed for robots; a broad variety of situations with rich semantics to understand and interpret; physical interactions with humans that requires fine, low-latency yet socially acceptable control strategies; natural and multi-modal communication which mandates common-sense knowledge and the representation of possibly divergent mental models. This article is an attempt to characterise these challenges and to exhibit a set of key decisional issues that need to be addressed for a cognitive robot to successfully share space and tasks with a human. We identify first the needed individual and collaborative cognitive skills: geometric reasoning and situation assessment based on perspective-taking and affordance analysis; acquisition and representation of knowledge models for multiple agents (humans and robots, with their specificities); situated, natural and multi-modal dialogue; human-aware task planning; human–robot joint task achievement. The article discusses each of these abilities, presents working implementations, and shows how they combine in a coherent and original deliberative architecture for human–robot interaction. Supported by experimental results, we eventually show how explicit knowledge management, both symbolic and geometric, proves to be instrumental to richer and more natural human–robot interactions by pushing for pervasive, human-level semantics within the robot's deliberative system.

Handing-over objects to humans (or taking objects from them) is a key capability for a service robot. Humans are efficient and natural while performing this action and the purpose of the studies on this topic is to bring human-robot handovers to an acceptable, efficient and natural level. This paper deals with the cues that allow to make a handover look as natural as possible, and more precisely we focus on where the robot should look while performing it. In this context we propose a user study, involving 33 volunteers, who judged video sequences where they see either a human or a robot giving them an object. They were presented with different sequences where the agents (robot or human) have different gaze behaviours, and were asked to give their feeling about the sequence naturalness. In addition to this subjective measure, the volunteers were equipped with an eye tracker which enabled us to have more accurate objective measures.

In daily human interactions, spatial reasoning occupies an important place. In this paper we present a situation assessment reasoner that generates relevant symbolic information from the geometry of the environment with respect to relations between objects and human capabilities. The role of SPARK (SPAtial Reasoning and Knowledge) component is to permanently maintain a state of the world in order to provide a basis for the robot to plan, to act, to react and to interact. More precisely, we describe here the way the system manages the hypotheses to be able to handle such knowledge in a flexible manner. Equipped with such capabilities, a robot that will interact with humans should be able to extract, compute or infer these relations and capabilities in order to communicate and interact efficiently in a natural way. To illustrate our work, we will explain how the robot is able to manage and update agents beliefs and pass Sally-Anne test. This work is part of a broader effort to develop a complete decisional framework for human-robot interactive task achievement.

Human Robot collaborative task achievement requires adapted tools and algorithms for both decision making and motion computation. The human presence as well as its behavior must be considered and actively monitored at the decisional level for the robot to produce synchronized and adapted behavior. Additionally, having a human within the robot range of action introduces security constraints as well as comfort considerations which must be taken into account at the motion planning and control level. This paper presents a robotic architecture adapted to human robot interaction and focuses on two tools: a human aware manipulation planner and a supervision system dedicated to collaborative task achievement.

Among the cognitive abilities a robot companion must be endowed with, human perception and speech understanding are both fundamental in the context of multimodal human-robot interaction. First, we propose a multiple object visual tracker which is interactively distributed and dedicated to two-handed gestures and head location in 3D. An on-board speech understanding system is also developed in order to process deictic and anaphoric utterances. Characteristics and performances for each of the two components are presented. Finally, integration and experiments on a robot companion highlight the relevance and complementarity of our multimodal interface. Outlook to future work is finally discussed.

Human-robot collaboration requires both communicative and decision making skills of a robot. To enable flexible coordination and turn-taking between human users and a robot in joint tasks, the robot's dialog and decision making mechanism have to be synchronized in a meaningful way. In this paper, we propose a integration framework to combine the dialog and the decision making processes. With this framework, we investigate various task negotiation situations for a social robot in a fetch-and-carry scenario. For the technical realization of the framework, the interface specification between the dialog and the decision making systems is also presented. Further, we discuss several challenging issues identified in our integration effort that should be adddressed in the future.

Human-robot collaborative task achievement requires the robot to reason not only about its current beliefs but also about the ones of its human partner. In this paper, we introduce a framework to manage shared knowledge for a robotic system dedicated to interactive task achievement with a human. In a first part, we define which beliefs should be taken into account ; we then explain a manner to achieve them using communication schemes. Several examples are presented to illustrate the purpose of beliefs management including a real experiment demonstrating a "give object" task between the Jido robotic platform and a human.

This paper addresses high-level robot planning issues for an interactive cognitive robot that has to act in presence or in collaboration with a human partner. We describe a task planner called HATP (for human aware task planner). HATP is especially designed to handle a set of human-centered constraints in order to provide "socially acceptable" plans that are oriented toward collaborative task achievement. We provide an overall description of HATP and discuss its main structure and algorithmic features.

Rackham is an interactive robot-guide that has been used in several places and exhibitions. This paper presents its design and reports on results that have been obtained after its deployment in a permanent exhibition. The project is conducted so as to incrementally enhance the robot functional and decisional capabilities based on the observation of the interaction between the public and the robot. Besides robustness and efficiency in the robot navigation abilities in a dynamic environment, our focus was to develop and test a methodology to integrate human-robot interaction abilities in a systematic way. We first present the robot and some of its key design issues. Then, we discuss a number of lessons that we have drawn from its use in interaction with the public and how that will serve to refine our design choices and to enhance robot efficiency and acceptability

The presence of humans in the robot environment brings new challenges to the robotic research. From low level functions to high level planners, clearly the human has to be taken into account in all the layers of the robot control system. Indeed, the robot has to behave socially in order to interact friendly with its human partners. This paper describes the development of several components (human detection and tracking, planning and supervision) that take into account humans explicitly with some preliminary results of their integration.