Motivated by the vision of integrating mobile robots closer to humans in warehouses, hospitals, manufacturing plants, and the home, we focus on robot navigation in dynamic and spatially constrained environments. Ensuring human safety, comfort, and efficiency in such settings requires that robots are endowed with a model of how humans move around them. Human motion prediction around robots is especially challenging due to the stochasticity of human behavior, differences in user preferences, and data scarcity. In this work, we perform a methodical investigation of the effects of human motion prediction quality on robot navigation performance, as well as human productivity and impressions. We design a scenario involving robot navigation among two human subjects in a constrained workspace and instantiate it in a user study (N=80) involving two different robot platforms, conducted across two sites from different world regions. Key findings include evidence that: 1) the widely adopted average displacement error is not a reliable predictor of robot navigation performance and human impressions; 2) the common assumption of human cooperation breaks down in constrained environments, with users often not reciprocating robot cooperation, and causing performance degradations; 3) more efficient robot navigation often comes at the expense of human efficiency and comfort.

Task and Motion Planning (TAMP) is a computationally challenging robotics problem due to the tight coupling of discrete symbolic planning and continuous geometric planning of robot motions. In particular, planning manipulation tasks in complex 3D environments leads to a large number of costly geometric planner queries to verify the feasibility of considered actions and plan their motions. To address this issue, we propose Geometric Reasoning Networks (GRN), a graph neural network (GNN)-based model for action and grasp feasibility prediction, designed to significantly reduce the dependency on the geometric planner. Moreover, we introduce two key interpretability mechanisms: inverse kinematics (IK) feasibility prediction and grasp obstruction (GO) estimation. These modules not only improve feasibility predictions accuracy, but also explain why certain actions or grasps are infeasible, thus allowing a more efficient search for a feasible solution. Through extensive experimental results, we show that our model outperforms state-of-the-art methods, while maintaining generalizability to more complex environments, diverse object shapes, multi-robot settings, and real-world robots.

Navigating in environments alongside humans requires agents to reason under uncertainty and account for the beliefs and intentions of those around them. Under a sequential decision-making framework, egocentric navigation can naturally be represented as a Markov Decision Process (MDP). However, social navigation additionally requires reasoning about the hidden beliefs of others, inherently leading to a Partially Observable Markov Decision Process (POMDP), where agents lack direct access to others’ mental states. Inspired by Theory of Mind and Epistemic Planning, we propose (1) a neuro-symbolic model-based reinforcement learning architecture for social navigation, addressing the challenge of belief tracking in partially observable environments; and (2) a perspective-shift operator for belief estimation, leveraging recent work on Influence-based Abstractions (IBA) in structured multi-agent settings.

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation . While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this article, we pave the road toward common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots, and datasets.

The hybrid discrete/continuous nature of task and motion planning (TAMP) results often in a combinatorial explosion. This challenge is even more pronounced in multi-robot TAMP problems due to the increase in dimensionality of the action space. Previous works use action feasibility prediction as a heuristic to accelerate TAMP. However, these methods are limited to box-shaped objects and specific single or dual robot settings. In this paper, we propose a feasibility-enabled multi-robot TAMP algorithm capable of tackling complex multi-robot manipulation problems. Also, we expand on our previous work on action and grasp feasibility prediction [1] by extending its use to mesh-shaped objects. We demonstrate the performance of our method compared to a non feasibility-informed baseline, and show its ability to handle TAMP problems requiring the collaboration of multiple robots.

Despite important progress in recent years, social robots are still far away from showing advanced behavior for interaction and adaptation in human environments. Thus, we are interested in studying social cognition in human-robot interaction (HRI), notably in improving communication skills relying on joint attention (JA) and knowledge sharing. Since JA involves low-level cognitive processes in humans, we take into account the implications of Moravec’s Paradox and focus on the aspect of knowledge representation. Inspired by 4E cognition principles, we study egocentric localization through the concept of sensory ego-sphere. We propose a neural network architecture named AEGO to model attention for each agent in interaction and show how to fuse information in a common representation space. From the perspective of dynamic fields theory, AEGO takes into account the dynamics of bottom-up and top-down modulation processes and the effects of neural excitatory and inhibitory synaptic interaction. In this work we evaluate the model in simulation and experiments with the robot Pepper in JA tasks based on proprioception, vision, rudimentary natural language and Hebbian plasticity. Results show that AEGO is convenient for HRI, allowing the human and the robot to share attention and knowledge about objects in scenarios close to everyday situations. AEGO constitutes a novel brain-inspired architecture to model attention that is suitable for multi-agent applications relying on social cognition skills, having the potential to generalize to several robotics platforms and HRI scenarios.

In Task and motion planning (TAMP), symbolic search is combined with continuous geometric planning. A task planner finds an action sequence while a motion planner checks its feasibility and plans the corresponding sequence of motions. However, due to the high combinatorial complexity of discrete search, the number of calls to the geometric planner can be very large. Previous works [1] [2] leverage learning methods to efficiently predict the feasibility of actions, much like humans do, on tabletop scenarios. This way, the time spent on motion planning can be greatly reduced. In this work, we generalize these methods to 3D environments, thus covering the whole workspace of the robot. We propose an efficient method for 3D scene representation, along with a deep neural network capable of predicting the probability of feasibility of an action. We develop a simple TAMP algorithm that integrates the trained classifier, and demonstrate the performance gain of using our approach on multiple problem domains. On complex problems, our method can reduce the time spent on geometric planning by up to 90%.

Coexisting with others and interacting in society implies sharing knowledge and attention about world objects, events, features, episodes, and even imagination or abstract ideas in time and space. Inspired by human phenomenological, cognitive and behavioral research, this work focuses on the study of joint attention (JA) for human-robot interaction (HRI), based on two main assumptions: a) the perception and representation of attention jointness constitute an isomorphic relation, and b) inspiration on dynamic neural fields (DNF) theory is a promising way to investigate contextual and non-linear spatio-temporal relations underlying attention and knowledge sharing in HRI. Taking into account the previous considerations, we propose a topology-based model for JA named TOP-JAM, which is able to represent and track in real-time JA states, from observations of behavioral data. More importantly, the model consists in a representation that can be directly understood by human beings, which conforms to robo-ethical principles in social robotics. This study evaluates computational properties of the model in simulation. Through a real experiment with the robot Pepper, the study shows that TOP-JAM is able to track JA in a triad interaction scenario.

In human-robot collaboration, the objectives of the human are often unknown to the robot. Moreover, even assuming a known objective, the human behavior is also uncertain. In order to plan a robust robot behavior, a key preliminary question is then: How to derive realistic human behaviors given a known objective? A major issue is that such a human behavior should itself account for the robot behavior, otherwise collaboration cannot happen. In this paper, we rely on Markov decision models, representing the uncertainty over the human objective as a probability distribution over a finite set of objective functions (inducing a distribution over human behaviors). Based on this, we propose two contributions: 1) an approach to automatically generate an uncertain human behavior (a policy) for each given objective function while accounting for possible robot behaviors; and 2) a robot planning algorithm that is robust to the above-mentioned uncertainties and relies on solving a partially observable Markov decision process (POMDP) obtained by reasoning on a distribution over human behaviors. A co-working scenario allows conducting experiments and presenting qualitative and quantitative results to evaluate our approach.

In this paper, we address task and motion plan-ning (TAMP) which is an important yet challenging robotics problem. It is known to suffer from the high combinatorial complexity of discrete search, often requiring a large number of geometric planning calls. We build upon recent works in TAMP by taking advantage of learning methods to provide action feasibility information as a heuristic to the symbolic planner, thus guiding it to a geometrically feasible solution and reducing geometric planning time. We propose AGFP-Net, a multi-task neural network predicting not only action feasibility, but also the feasibility of a set of grasp types. We also propose an improved feasibility-informed TAMP algorithm capable of solving more complex problems, and handling goals which are not fully specified. Comparative results obtained on different problems of varying complexity show that our method is able to greatly reduce task and motion planning time.

It is essential for a collaborative robot to consider the Theory of Mind (ToM) when interacting with humans. Indeed, performing an action in the absence of another agent may create false beliefs like in the well-known Sally & Anne Task [1]. The robot should be able to detect, react to, and even anticipate false beliefs of other agents with a detrimental impact on the task to achieve. Currently, ToM is mainly used to control the task execution and resolve in a reactive way the detrimental false beliefs. Some works introduce ToM at the planning level by considering distinct beliefs, and we are in this context. This work proposes an extension of an existing human-aware task planner and effectively allows the robot to anticipate a false human belief ensuring a smooth collaboration through an implicitly coordinated plan. First, we propose to capture the observability properties of the environment in the state description using two observability types and the notion of co-presence. They allow us to maintain distinct agent beliefs by reasoning directly on what agents can observe through specifically modeled Situation Assessment processes, instead of reasoning of action effects. Then, thanks to the better estimated human beliefs, we can predict if a false belief with adverse impact will occur. If that is the case then, first, the robot’s plan can be to communicate minimally and proactively. Second, if this false belief is due to a non-observed robot action, the robot’s plan can be to postpone this action until it can be observed by the human, avoiding the creation of the false belief. We implemented our new conceptual approach, discuss its effectiveness qualitatively, and show experimental results on three novel domains.

Human-aware robot navigation planning enables robots to traverse human-occupied spaces socially. However, evaluating and benchmarking the ‘human awareness’ of such navigation schemes is challenging. With the growing necessity and research interest in the field, there is a need to define metrics to quantify and benchmark such qualities. In this regard, this paper proposes a set of metrics by looking at the problem from a new perspective. These proposals are made by inspecting the robot’s navigation from the viewpoint of a human experiencing it and then defining proxies for the perceived human feelings. Analyses of some commonly occurring human-robot navigation scenarios using these metrics show their capability in benchmarking and differentiating human-aware robot navigation from standard robot navigation.

Navigation of robots among humans is still an open problem, especially in confined locations (e.g. narrow corridors, doors). This article aims at finding how an anthropomorphic robot, like a PR2 robot with a height of 1.33 m, should behave when crossing a human in a narrow corridor in order to increase its usability. Two experiments studied how a combination of robot head behavior and navigation strategy can enhance robot legibility. Experiment 1 aimed to measure where a pedestrian looks when crossing another pedestrian, comparing the nature of the pedestrian: human or a robot. Based on the results of this experiment and the literature, we then designed a robot behavior exhibiting mutual manifestness by both modifying its trajectory to be more legible, and using its head to glance at the human. Experiment 2 evaluated this behavior in real situations of pedestrians crossing a robot. The visual behavior and user experience of pedestrians were assessed. The first experiment revealed that humans primarily look at the robot's head just before crossing. The second experiment showed that when crossing a human in a narrow corridor, both modifying the robot trajectory and glancing at the human is necessary to significantly increase the usability of the robot. We suggest using mutual manifestness is crucial for an anthropomorphic robot when crossing a human in a corridor. It should be conveyed both by altering the trajectory and by showing the robot awareness of the human presence through the robot head motion. Small changes in robot trajectory and manifesting robot perception of the human via a user identified robot head can avoid users' hesitation and feeling of threat.

Experimenting, testing, and debugging robot social navigation systems is a challenging task. While simulation is generally well suited for a first level of debugging and evaluation of robotics controllers and planners, the social navigation field lacks satisfactory simulators of humans which act, react and interact rationally and naturally. To facilitate the development of human-aware navigation systems, we propose a system to simulate an autonomous human agent that is both reactive and rational, specifically designed to act and interact with a robot for navigation problems and potential conflicts. Besides, it also provides some metrics to partially evaluate such interactions and data logs for further analysis. We show the limitations of over-used reactive-only approaches. Then, thanks to two different human-aware navigation planners, we show how our system can help answer the lack of intelligent human avatars for tuning and debugging social navigation systems before their final evaluation with real humans.

Research in joint action focuses on the psychological, neurological, and physical mechanisms by which humans collabo-rate with other agents, and overlaps with several domains related to human-robot interaction. The development of artificial systems that can support or emulate the requisite aspects of joint action could lead to improved human-robot team performance as well as improvements in subjective metrics (e.g., trust). This workshop highlights theoretical and technical considerations about human-robot joint action and real-time adaptation, with a particular focus on socio-motor entrainment, showing how the emulation of psychological mechanisms (e.g., emotion, intention signaling, mirroring) can lead to improved performance. We will invite speakers with backgrounds in robotics, neuroscience and psychol-ogy, as well as speakers with a focus in adjacent works, such as in human-robot coordinated dance, alignment, or synchronization. We will call for papers that utilize the theory of joint-action in an interactive human-robot context. We will also call for position papers on the application of the theory of joint action to robotics, with a heavy focus on psychological mechanisms that could potentially be emulated or adapted to a human-robot context. Participants will have the opportunity to brainstorm considerations and techniques that would be applicable to joint action inspired works through breakout sessions with the aim to lead to new and improved collaborations across fields.

The variety and complexity of tasks autonomous robots can tackle is constantly increasing, yet we seldom see robots collaborating with humans. Indeed, humans are either requested for punctual help or are given the lead on the whole task. We propose a human-aware task planning approach allowing the robot to plan for a task while also considering and emulating the human decision, action, and reaction processes. Our approach, named Human-Aware Task Planner with Emulation of Human Decisions and Actions (HATP/EHDA), is based on the exploration of multiple hierarchical tasks networks albeit differently whether the agent is considered to be controllable (the robot) or uncontrollable (the human). We present the rationale of our approach along with a formalization and show its potential on an illustrative example.

The use of drones in human-populated areas is increasing day by day. Such robots flying in close proximity to humans and potentially interacting with them, as in object handover or delivery, need to carefully plan their navigation considering the presence of humans. We propose a humanaware 3D reactive planner based on stochastic optimization for drone navigation. Besides considering the kinematics constraints of the drone, we propose two criteria to produce socially acceptable trajectories. The first, called discomfort, considers the unease caused to the humans spatially close to fast-moving drones. The second, called visibility, promotes the drone's visibility for humans. We demonstrate the planner's performance and adaptability in various simulated experiments.

Current approaches in human-aware or social robot navigation address the humans that are visible to the robot. However, it is also important to address the possible emergences of humans to avoid shocks or surprises to humans and erratic behavior of the robot planner. In this paper, we propose a novel approach to detect and address these human emergences called ‘invisible humans’. We determine the places from which a human, currently not visible to the robot, can appear suddenly and then adapt the path and speed of the robot with the anticipation of potential collisions. This is done while still considering and adapting humans present in the robot's field of view. We also show how this detection can be exploited to identify and address the doorways or narrow passages. Finally, the effectiveness of the proposed methodology is shown through several simulated and real-world experiments.

The supervision component is the binder of a robotic architecture. Without it, there is no task, no interaction happening, it conducts the other components of the architecture towards the achievement of a goal, which means, in the context of a collaboration with a human, to bring changes in the physical environment and to update the human partner mental state. However, not so much work focus on this component in charge of the robot decision-making and control, whereas this is the robot puppeteer. Most often, either tasks are simply scripted, or the supervisor is built for a specific task. Thus, we propose JAHRVIS, a Joint Action-based Human-aware supeRVISor. It aims at being task-independent while implementing a set of key joint action and collaboration mechanisms. With this contribution, we intend to move the deployment of autonomous collaborative robots forward, accompanying this paper with our open-source code.

Being able to refer to an object, a person, or a place in a non-ambiguous manner is a need when one has to achieve collaborative activities with a partner. This is the so-called Referring Expression Generation (REG) problem. While widely used for Human-Robot Interaction, state of the art approaches restrict its use to the current environment. We propose a novel extension to the REG which takes full advantage of the Human-Robot shared knowledge about past actions as additional information to generate Referring Expressions. We show that our approach is usable with a domain-independent ontology as a knowledge base and that it can also use a semantic representation of past activity to generate RE. We illustrate our method through simulated situations and discuss its efficiency and pertinence.

As more robots are being deployed into human environments, a human-aware navigation planner needs to handle multiple contexts that occur in indoor and outdoor environments. In this paper, we propose a tunable human-aware robot navigation planner that can handle a variety of human-robot contexts. We present the architecture of the system and discuss the features along with some implementation details. Then we present a detailed analysis of various simulated human-robot contexts using the proposed planner. Further, we show that our system performs better when compared with an exiting human-aware planner in various contexts. Finally, we show the results in a real-world scenario after deploying our system on a real robot.

Today, physical human-robot interaction (pHRI) is a very popular topic in the field of ground manipulation. At the same time, aerial physical interaction is also developing very fast. Nevertheless, pHRI with aerial vehicles has not been addressed so far. In this work, we present the study of one of the first systems in which a human is physically connected to an aerial vehicle by a cable. We want the robot to be able to pull the human toward a desired position (or along a path) only using forces as an indirect communication-channel. We propose an admittance-based approach with a controller, inspired by the literature on flexible manipulators, that computes the desired interaction forces that properly guide the human. The stability of the system is formally proved with a Lyapunov-based argument. The system is also shown to be passive, and thus robust to nonidealities like model and tracking errors, additional human forces, time-varying inputs, and other external disturbances. We also design a maneuver regulation policy to simplify the path following problem. The global method has been experimentally validated on a group of four subjects, showing a reliable and safe pHRI.

In Human-Robot Interaction (HRI), ensuring nonambiguous communication between the robot and the human is a key point for carrying out fluently a collaborative task. With this work, we propose a method which allows the robot to generate the optimal set of assertions that are necessary in order to produce an unambiguous reference. In this paper, we present a novel approach to the Referring Expression Generation (REG) problem and its integration into a robotic system. Our method is a domain-independent approach based on an ontology as a knowledge base. We show how this generation can be performed on an ontology which is not dedicated to this task. We then validate our method through simulated situations, compare it with state of the art approach and on a real robotic system.

When we perform a collaborative task with another human, we are able to tell, to a certain extent, how things are going and more precisely if things are going well or not. This knowledge allows us to adapt our behavior. Therefore, we think it is desirable to provide robots with means to measure in real-time the Quality of the Interaction with their human partners. To make this possible, we propose a model and a set of metrics targeting the evaluation of the QoI in collaborative tasks through the measure of the human engagement and the online task effectiveness. These model and metrics have been implemented and tested within the high-level controller of an entertainment robot deployed in a mall. The first results show significant differences in the computed QoI when in interaction with a fully compliant human, a confused human and a non-cooperative one.

We propose a new framework combining decision making and planning in the human-robot co-navigation scenario. This new framework, called HATEB-2, introduces different modalities of planning and shift between them based on the situation at hand. These transitions are controlled by the decision making loop present on top of the planning. We also present the improvements made to human prediction and estimation along with the modifications to a few social constraints from our previous work, that are included in HATEB-2. Finally, several experiments are performed in human-robot co-navigation scenarios and results are presented. One of the modalities of HATEB-2 is used in EU-funded MuMMER [1] project (http://mummer-project.eu/).

This paper presents an innovative architecture of help desk in the tourism environment. The aim of the work, performed with a professional woodworker manufacturer, is to design an interactive counter, integrating the methods and the technologies of robotics. The idea is to provide customers with a new experience, based on an intelligent and social interaction within a connected environment, using wood as the main material. In order to achieve this objective, modular hardware and software architectures have been designed and the first prototype is under development. The paper focuses on the description of the architecture of this prototype and on the first results obtained.

Reasoning about spatial and geometric relations between objects in a tabletop human-robot interaction is a challenge due to the perception not being always consistent: objects placed on a table seem to be slightly in the air; they overlap; they disappear due to occlusions. Yet, interpreting and anchoring perceptual data in a physically consistent estimation of the scene is a crucial ability for humans, and thus robots in HRI context. In this paper we present a simulation-based physics reasoner integrated in a lightweight situation-assessment framework called Underworlds, that allows the robot to stabilize objects and build at run-time a consistent estimation of the scene, even for entirely hidden objects, while inferring the actions performed by its human partner.

In this paper we present Ontologenius, a semantic knowledge storage and reasoning framework for autonomous robots. More than a classic ontology software to query a knowledge base and a first-order internal logic as it can be done for web-semantics, we propose with Ontologenius features adapted to a robotic use including human-robot interaction. We introduce the ability to modify the knowledge base during execution, whether through dialogue or geometric reasoning, and keep these changes even after the robot is powered off. Since Ontologenius was developed to be used by a robot which interacts with humans, we have endowed the system with ability to perform attributes and properties generalization and with the possibility to model and estimate the semantic memory of a human partner and to implement theory of mind processes. This paper presents the architecture and the main features of Ontologenius as well as examples of its use in robotics applications.

We claim that the activity consisting in providing route directions can be best dealt with as a joint task involving the contribution not only of the robot as a direction provider but also of the human as listener. Moreover, we claim that in some cases, both the robot and the human should move to reach a different perspective of the environment which allows the explanations to be more efficient. As a first step toward implementing such a system, we propose the SVP (Shared Visual Perspective) planner which searches for the right placements both for the robot and the human to enable the visual perspective sharing needed for providing route direction and which makes the choice of the best landmark when several are available. The shared perspective is chosen taking into account not only the visibility of the landmarks, but the whole guiding task.

We introduce UNDERWORLDS, a novel lightweight framework for cascading spatio-temporal situation assessment in robotics. UNDERWORLDS allows programmers to represent the robot's environment as real-time distributed data structures, containing both scene graphs (for representation of 3D geometries) and timelines (for representation of temporal events). UNDERWORLDS supports cascading representations: the environment is viewed as a set of worlds that can each have different spatial and temporal granularities, and may inherit from each other. UNDERWORLDS also provides a set of high-level client libraries and tools to introspect and manipulate the environment models. This article presents the design and architecture of this open-source tool, and explores some applications, along with examples of use.

The domain of human-robot Joint Action is a growing field where roboticists, psychologists and philosophers start to collaborate in order to devise robot abilities that are as efficient and convenient for the human partner as possible. Besides studying Joint Action and developing algorithms and schemes to control the robot and manage the interaction, one of the current challenges is to come up with a method to properly evaluate the progresses made by the community. Several questionnaires have already been proposed to the community that deal with the evaluation of human-robot interaction. However, these studies mainly concern either specific basic behaviors during Joint Action or human-robot interactions without effective physical Joint Action. When it comes to high level decisions during physical human-robot Joint Action, there are fewer contributions to the topic, and also, the methods to evaluate them are even rarer. The aim of this paper is to propose a reusable questionnaire PeRDITA (Pertinence of Robot Decisions In joinT Action) allowing us to evaluate the pertinence of high level decision abilities of a robot during physical Joint Action with a human.

This paper focuses on requirements for effective human robot collaboration in interactive navigation scenarios. We designed several use-cases where humans and robot had to move in the same environment that resemble canonical path-crossing situations. These use-cases include open as well as constrained spaces. Three different state-of-the-art human-aware navigation planners were used for planning the robot paths during all selected use-cases. We compare results of simulation experiments with these human-aware planners in terms of quality of generated trajectories together with discussion on capabilities and limitations of the planners. The results show that the human-robot collaborative planner [1] performs better in everyday path-crossing configurations. This suggests that the criteria used by the human-robot collaborative planner (safety, time-to-collision, directional-costs) are possible good measures for designing acceptable human-aware navigation planners. Consequently, we analyze the effects of these social criteria and draw perspectives on future evolution of human-aware navigation planning methods.

Human–Robot Interaction challenges Artificial Intelligence in many regards: dynamic, partially unknown environments that were not originally designed for robots; a broad variety of situations with rich semantics to understand and interpret; physical interactions with humans that requires fine, low-latency yet socially acceptable control strategies; natural and multi-modal communication which mandates common-sense knowledge and the representation of possibly divergent mental models. This article is an attempt to characterise these challenges and to exhibit a set of key decisional issues that need to be addressed for a cognitive robot to successfully share space and tasks with a human. We identify first the needed individual and collaborative cognitive skills: geometric reasoning and situation assessment based on perspective-taking and affordance analysis; acquisition and representation of knowledge models for multiple agents (humans and robots, with their specificities); situated, natural and multi-modal dialogue; human-aware task planning; human–robot joint task achievement. The article discusses each of these abilities, presents working implementations, and shows how they combine in a coherent and original deliberative architecture for human–robot interaction. Supported by experimental results, we eventually show how explicit knowledge management, both symbolic and geometric, proves to be instrumental to richer and more natural human–robot interactions by pushing for pervasive, human-level semantics within the robot's deliberative system.

When a robot has to execute a shared plan with a human, a number of unexpected situations and contingencies can happen due, essentially, to human initiative. For instance, a temporary absence or inattention of the human can entail a partial, and potentially not sufficient, knowledge about the current situation. To ensure a successful and fluent execution of the shared plan the robot might need to detect such situations and be able to provide the information to its human partner about what he missed without being annoying or intrusive. To do so, we have developed a framework which allows a robot to estimate the other agents mental states not only about the environment but also about the state of goals, plans and actions and to take them into account when executing human-robot shared plans.

One application of robotics is to assist humans in the achievement of tasks they face in both the workplace and domestic environments. In some situations, a task may require the robot and the human to act together in a collaborative way in order to reach a common goal. To achieve a collaborative plan, each agent (human, robot) needs to be aware of the tasks she/he must carry out and how to perform them. This paper addresses the issue of enhancing a robotic system with a dynamic model of its collaborator's knowledge concerning tasks of a shared plan. Using this model, the robot is able to adapt its collaborative plan generation, its abilities to give explanations and to monitor the overall plan execution. We present the algorithm we have elaborated to take advantage of the tree representation of our Hierarchical Task Network (HTN) planner to enhance the robot with appropriate explanation and execution monitoring abilities. To evaluate how our adaptive system is perceived by users and how much it improves the quality of the Human-Robot interaction, the outcome of a comparative study is presented.

For intelligent Human-Robot Interaction (HRI), a robot should be equipped with some core reasoning capabilities such as perspective taking, effort analysis, and affordance analysis. This paper starts to explore how a robot equipped with such reasoning abilities could be evaluated. To this end, inspired by the Turing test, we design a game involving a human-robot competition scenario. Interestingly, the participants' subjective feedback, which tended to compare the robot's abilities with their own, points toward potential criteria for developing benchmark scenarios and evaluation matrices.

A human-robot interaction system should be capable of adapting the execution of cooperative plans with respect to complex human activities and interventions. In this paper, we present an integrated framework that exploits attentional supervision and contention scheduling to combine human-aware planning, plan execution, and natural human-robot interaction. Specifically, in the proposed approach, hierarchical cooperative plans are exploited as top-down attentional guidance for the robotic executive system, which can flexibly orchestrate the task activities while reacting to environmental stimuli and human behaviors. We describe the overall framework discussing some case studies in human-robot collaborative scenarios.

Robot navigation in human environments has been in the eyes of researchers for the last few years. Robots operating under these circumstances have to take human awareness into consideration for safety and acceptance reasons. Nonetheless, navigation have been often treated as going towards a goal point or avoiding people, without considering the robot engaging a person or a group of people in order to interact with them. This paper presents two navigation approaches based on the use of inverse reinforcement learning (IRL) from exemplar situations. This allow us to implement two path planners that take into account social norms for navigation towards isolated people. For the first planner, we learn an appropriate way to approach a person in an open area without static obstacles, this information is used to generate robot's path plan. As for the second planner, we learn the weights of a linear combination of continuous functions that we use to generate a costmap for the approach-behavior. This costmap is then combined with others, e.g. a costmap with higher cost around obstacles, and finally a path is generated with Dijkstra's algorithm.

The overall performance of sampling-based motion planning algorithms strongly depends on the use of suitable sampling and connection strategies, as well as on the accuracy of the distance metric considered to select neighbor states. Defining appropriate strategies and metrics is particularly hard when considering robot dynamics, which is required to treat constrained motion planning problems for quadrotors. This paper presents an accurate but computationally fast quasi-metric to determine the proximity of dynamic states of a quadrotor, and an incremental state-space sampling technique to avoid generating local trajectories that violate kinodynamic constraints. Results show that the integration of the proposed techniques in RRT-based and PRM-based algorithms can drastically decrease computing time, up to two orders of magnitude.

We are combining symbolic and geometric planning to synthesize human-aware plans in order to deal with the complex and highly intricate planning problems induced by Human-Robot collaborative object manipulation. In this paper, we summarize our previous contributions - refining symbolic actions at geometric level, during the symbolic planning, in order to assess their feasibility and computing the geometric side effects-, then we present the current contributions meant to tighten the cooperation between the symbolic planning and the geometric planning: the symbolic planner helps the geometric one by providing it with constraints and domain-expert knowledge making the geometric planner more efficient, and the geometric planner helps the symbolic one to find the best plan based on social costs computed at geometric level. We also propose different examples, highlighting the interest of such cooperation between the planners in simulation and on our PR2 robot.

HTN planners have generally relied on specialised languages for domain and problem representations. To facilitate adoption by other communities such as robotics, however, and integration with real world applications written in standard programming languages, we need HTN planners that are based on more familiar concepts from structured programming, and that come ready with features supporting integration. In this paper, we demonstrate how the HATP (Hierarchical Agent-based Task Planner) HTN planner offers such “syntactic sugar” and some of these features. Moreover, since it has a conceptually distinct syntax compared to traditional HTN planners, we also develop a formalism to unambiguously capture HATP's syntax and an important subset of its semantics, which we then use to compare against the formalism of a well understood family of HTN planners and to show that the former is sound. Finally, we demonstrate that despite quite possibly using “heavier” data structures to naturally capture HATP's syntax/semantics, and thereby facilitate extensions to HATP and integration with other applications, the implementation still performs acceptably.

Exchanging objects with humans through hand-overs is a key feature for any robot operating side by side with humans. The studies on the topic tackle various problems, such as the hand dexterity, the communication cues, the arms motions, the forces, the head motions, but most of them consider a handover as an independent action, decorrelated from the plan it is part of.

Handing-over objects to humans (or taking objects from them) is a key capability for a service robot. Humans are efficient and natural while performing this action and the purpose of the studies on this topic is to bring human-robot handovers to an acceptable, efficient and natural level. This paper deals with the cues that allow to make a handover look as natural as possible, and more precisely we focus on where the robot should look while performing it. In this context we propose a user study, involving 33 volunteers, who judged video sequences where they see either a human or a robot giving them an object. They were presented with different sequences where the agents (robot or human) have different gaze behaviours, and were asked to give their feeling about the sequence naturalness. In addition to this subjective measure, the volunteers were equipped with an eye tracker which enabled us to have more accurate objective measures.

A common approach to social distancing in robot navigation are spatial cost functions around humans that cause the robot to prefer paths that do not come too close to humans. However, in unpredictably dynamic scenarios, following such paths may produce robot behavior that appears confused. The concept of directional costs in cost functions [9] is supposed to alleviate this problem without incurring the problem of combinatorial explosions using temporal planning. With directional cost functions, a robot attempts to solve spatial conflicts by adjusting the velocity instead of the path, where possible. To complement results from simulations, in this paper we describe a user study we conducted with a PR2 robot and human participants to evaluate the new cost function type. The study shows that the real robot behavior is similar to the observations in simulation, and that participants rate the robot behavior less confusing with the adapted cost model. The study also shows other important behavior cues that can influence motion legibility.

Bridging the gap between symbolic and geometric planning has received much attention in recent years. An important issue in some of the works that combine the two approaches is finding the right balance between backtracking at the symbolic level versus at the geometric planning level. We present in this work a new approach to interleaved backtracking, where the symbolic planner backtracks to try alternative action branches that naturally map to different geometric solutions. This eliminates the need to “protect” certain symbolic conditions when backtracking at the geometric level, and addresses a completeness issue in our previous approach to interleaved backtracking. We discuss a concrete, non-trivial symbolic-geometric planning example in the context of Human-Robot Interaction, a full implementation of the combined planning technique, and an evaluation of performance as well as the effect of increasing the symbolic-action branching factor.

Planning physically realistic and easily controllable motions of flying robots requires considering dynamics. This paper presents a local trajectory planner, based on a simplified dynamic model of quadrotors, which fits the requirements to be integrated into a global motion planning approach. It relies on a closed-form solution to compute curves in the kinodynamic state space that tend to minimize the flying time. These curves have suitable continuity properties and guarantee respect of physical limits of the system (i.e. bounds for the time-derivatives of the pose coordinates). The paper explains how this local planner can be used within different motion planning approaches that enable the treatment of difficult problems in constrained environments.

In daily human interactions, spatial reasoning occupies an important place. In this paper we present a situation assessment reasoner that generates relevant symbolic information from the geometry of the environment with respect to relations between objects and human capabilities. The role of SPARK (SPAtial Reasoning and Knowledge) component is to permanently maintain a state of the world in order to provide a basis for the robot to plan, to act, to react and to interact. More precisely, we describe here the way the system manages the hypotheses to be able to handle such knowledge in a flexible manner. Equipped with such capabilities, a robot that will interact with humans should be able to extract, compute or infer these relations and capabilities in order to communicate and interact efficiently in a natural way. To illustrate our work, we will explain how the robot is able to manage and update agents beliefs and pass Sally-Anne test. This work is part of a broader effort to develop a complete decisional framework for human-robot interactive task achievement.

Over the last four years, we have been slowly ramping up explicit knowledge representation and manipulation in the deliberative and executive layers of our robots. Ranging from situation assessment to symbolic task planning, from verbal interaction to event-driven execution control, we have built up a knowledge-oriented architecture which is now used on a daily basis on our robots. This article presents our design choices, the articulations between the diverse deliberative components of the robot, and the strengths and weaknesses of this approach. We show that explicit knowledge management is not only a convenient tool from the software engineering point of view, but also pushes for a different, more semantic way to address the decision-making issue in autonomous robots.

Analyzing affordances has its root in socio-cognitive development of primates. Knowing what the environment, including other agents, can offer in terms of action capabilities is important for our day-to-day interaction and cooperation. In this paper, we will merge two complementary aspects of affordances: from agent-object perspective, what an agent afford to do with an object, and from agent-agent perspective, what an agent can afford to do for other agent, and present a unified notion of Affordance Graph. The graph will encode affordances for a variety of tasks: take, give, pick, put on, put into, show, hide, make accessible, etc. Another novelty will be to incorporate the aspects of effort and perspective-taking in constructing such graph. Hence, the Affordance Graph will tell about the action-capabilities of manipulating the objects among the agents and across the places, along with the information about the required level of efforts and the potential places. We will also demonstrate some interesting applications.

While symbolic planners work with an abstract representation of the real world, allowing plans to be constructed relatively quickly, geometric planning - although more computationally complex - is essential for building symbolic plans that actually work in the real world. To combine the two types of systems, we present in this paper a meaningful interface, and insights into a methodology for developing interwoven symbolic-geometric domains. We concretely present this “link” between the two approaches with algorithms and data structures that amount to an intermediate layer that coordinates symbolic-geometric planning. Since both planners are capable of “backtracking” at their own levels, we also investigate the issue of how to interleave their backtracking, which we do in the context of the algorithms that form the link. Finally, we present a prototype implementation of the combined system on a PR2 robot.

Over the last two years, the Modular OpenRobots Simulation Engine (MORSE) project1 went from a simple extension plugged on the Blender's Game Engine to a full-fledged simulation environment for academic robotics. Driven by the requirements of several of its developers, tools dedicated to Human-Robot interaction simulation have taken a prominent place in the project. This late breaking report discusses some of the recent additions in this domain, including the immersive experience provided by the integration of the Kinect device as input controller. We also give an overview of the experiences we plan to complete in the coming months.

No abstract available.

For a versatile human-assisting mobile-manipulating robot such as the PR2, handing over objects to humans in possibly cluttered workspaces is a key capability. In this paper we investigate the motion planning of handovers while accounting for the human mobility. We treat the human motion as part of the planning problem thus enabling to find broader type of handing strategies. We formalize the problem and propose an algorithmic solution taking into account the HRI constraints induced by the human receiver presence. Simulation results with the PR2 robot illustrate the efficacy of the approach.

Affordance analysis, what something/someone can afford or offers, is an important aspect for day-to-day interaction and decision-making. In this paper, we will enrich the notion of affordance by incorporating agent-agent affordance: what does an agent afford for another agent in terms of a task. Further, we will present an effort hierarchy and derive the concept of Taskability Graph, which encodes: what all agents could do for all other agents, with which levels of mutual-efforts and at which places. This makes the robot more aware about agents' abilities and facilitates to develop better interaction and decision-making capabilities. We will discuss the potential application in effort based shared cooperative planning.

We have designed and implemented new spatio-temporal reasoning skills for a cognitive robot, which explicitly reasons about human beliefs on object positions. It enables the robot to build symbolic models reflecting each agent's perspective on the world. Using these models, the robot has a better understanding of what humans say and do, and is able to reason on what human should know to achieve a given goal. These new capabilities are also demonstrated experimentally.

With recent advances in safe and compliant hardware and control, robots are close to finding their places in our homes. As the safety barrier between humans and robots is beginning to fade, the necessity to design pertinent robot behavior in human environments is becoming a crucial step. In order to obtain a safe, comfortable, and socially acceptable interaction, the robot should be engineered from top to bottom by considering the presence of the human. In this paper, we present a manipulation planning framework and its implementation human-aware manipulation planner. This planner generates paths not only safe but comfortable and “socially acceptable” as well by reasoning explicitly on human's kinematics, vision field, posture, and preferences. The planner, which is applied into “robot handing over an object” scenarios, breaks the human centric interaction that depends mostly on human effort and allows the robot to take initiative by computing automatically where the interaction takes place, thus decreasing the cognitive weight of interaction on human side.

This paper addresses the motion planning problem while considering Human-Robot Interaction (HRI) constraints. The proposed planner generates collision-free paths that are acceptable and legible to the human. The method extends our previous work on human-aware path planning to cluttered environments. A randomized cost-based exploration method provides an initial path that is relevant with respect to HRI and workspace constraints. The quality of the path is further improved with a local path-optimization method. Simulation results on mobile manipulators in the presence of humans demonstrate the overall efficacy of the approach.

If robots are to cooperate with humans in an increasingly human-like manner, then significant progress must be made in their abilities to observe and learn to perform novel goal directed actions in a flexible and adaptive manner. The current research addresses this challenge. In CHRIS.I [1], we developed a platform-independent perceptual system that learns from observation to recognize human actions in a way which abstracted from the specifics of the robotic platform, learning actions including “put X on Y” and “take X”. In the current research, we extend this system from action perception to execution, consistent with current developmental research in human understanding of goal directed action and teleological reasoning. We demonstrate the platform independence with experiments on three different robots. In Experiments 1 and 2 we complete our previous study of perception of actions “put” and “take” demonstrating how the system learns to execute these same actions, along with new related actions “cover” and “uncover” based on the composition of action primitives “grasp X” and “release X at Y”. Significantly, these compositional action execution specifications learned on one iCub robot are then executed on another, based on the abstraction layer of motor primitives. Experiment 3 further validates the platform-independence of the system, as a new action that is learned on the iCub in Lyon is then executed on the Jido robot in Toulouse. In Experiment 4 we extended the definition of action perception to include the notion of agency, again inspired by developmental studies of agency attribution, exploiting the Kinect motion capture system for tracking human motion. Finally in Experiment 5 we demonstrate how the combined representation of action in terms of perception and execution provides the basis for imitation. This provides the basis for an open ended cooperation capability where new actions can be learned and integrated into shared plans for cooperation. Part of the novelty of this research is the robots' use of spoken language understanding and visual perception to generate action representations in a platform independent manner based on physical state changes. This provides a flexible capability for goal-directed action imitation.

While key for human-robot interaction, natural language interpretation is a notoriously difficult task, especially because the interaction context is at the same time essential for dialogue understanding, difficult to build for machines, and depends on each speaker point of view. However, robots as embodied artifacts, can perceive their environment and interactors, and hence compute symbolic models from various perspectives. This allows in turn to build symbolic contexts for dialogues. In this paper, we introduce DIALOGS, a component for natural language interpretation that relies on these structured symbolic models of the world to ground verbal interaction.

In daily human interactions spatial reasoning occupies an important place. With this ability we can build relations between objects and people, and we can predict the capabilities and the knowledge of the people around us. An interactive robot is also expected to have these abilities in order to establish an efficient and natural interaction. In this paper we present a situation assessment reasoner, based on spatial reasoning and perspective taking, which generates on-line relations between objects and agents in the environment. Being fully integrated to a complete architecture, this reasoner sends the generated symbolic knowledge to a fact data base which is built on the basis on an ontology and which is accessible to the entire system. This work is also part of a broader effort to develop a complete decisional framework for human-robot interactive task achievement.

Humans constantly generate and solve ambiguities while interacting with each other in their every day activities. Hence, having a robot that is able to solve ambiguous situations is essential if we aim at achieving a fluent and acceptable human-robot interaction. We propose a strategy that combines three mechanisms to clarify ambiguous situations generated by the human partner. We implemented our approach and successfully performed validation tests in several different situations both, in simulation and with the HRP-2 robot.

This paper presents an embeddable knowledge processing framework, along with a common-sense ontology, designed for robotics. We believe that a direct and explicit integration of cognition is a compulsory step to enable human-robots interaction in semantic-rich human environments like our houses. The OpenRobots Ontology (ORO) kernel allows to turn previously acquired symbols into concepts linked to each other. It enables in turn reasoning and the implementation of other advanced cognitive functions like events, categorization, memory management and reasoning on parallel cognitive models. We validate this framework on several cognitive scenarii that have been implemented on three different robotic architectures.

Interestingly Humans are able to maintain rough estimations of visibility, reachability and other capabilities of not only themselves but of the person they are interacting with. Studies in neuroscience and psychology suggest that from the age of 12-15 months children start to understand the occlusion of others line-of-sight and from the age of 3 years they start to develop the ability, termed as perceived reachability for self and for others. As such capabilities evolve in the children, they start showing intuitive and proactive behavior by perceiving various abilities of the human partner. Inspired from such studies, which suggest that visuo-spatial perception plays an important role in Human-Human interaction, we propose to equip our robot with the capabilities to maintain various types of reachabilities and visibilities information of itself and of the human partner in the shared workspace. Since these analyses will be basically perceived by performing a virtual action onto the agent and roughly estimating what that agent might be able to 'see' and 'reach' in 3D space, we term these representations as Mightability Maps. By applying various set operations on Weighted Mightability Maps, robot could perceive a set of candidate solutions in real time for various tasks. We show its application in exhibiting two different behaviors of robot: co-operative and competitive. These maps are also quick to compute and could help in developing higher-level decisional capabilities in the robot.

For a Mobile Robot to navigate in the Human-Centered environment without imposing alien like impression by its motion, it should be able to reason about various criteria ranging from clearance, environment structure, unknown objects, social conventions, proximity constraints, presence of an individual or group of peoples, etc. Also the robot should neither be over-reactive nor be simple wait and move machine. We have adapted a Voronoi diagram based approach for the analysis of local clearance and environment structure. We also propose to treat human differently from other obstacles for which the robot constructs different sets of regions around human and iteratively converges to a set of points (milestones), using social conventions, human proximity guidelines and clearance constraints to generate and modify its path smoothly. Once equipped with such capabilities, robot is able to do higher-level reasoning for dynamic and selective adaptation of social convention depending upon the environment segment. It also leads the robot to be aware about its own motion behavior.

In human-robot interaction, a robot must be prepared to handle possible ambiguities generated by a human partner. In this work we propose a set of strategies that allow a robot to identify the referent when the human partner refers to an object giving incomplete information, i.e. an ambiguous description. Moreover, we propose the use of an ontology to store and reason on the robot's knowledge to ease clarification, and therefore, improve interaction. We validate our work through both simulation and two real robotic platforms performing two tasks: a daily-life situation and a game.

Robot path planning has traditionally concentrated on collision-free paths. For robots that collaborate closely with humans, however, the situation is different in two respects: 1) the humans in the robot's environment are not randomly moving objects, but cognitive beings who can deliberately make way for a robot to pass and 2) the quality of a navigation plan depends less on quantitative efficiency criteria, but rather on the acceptance of humans. In this paper, we introduce a robot navigation approach that takes into account human-centered requirements and the collaborative nature of the interaction between the human and the robot.

A method of regrasp planning for humanoid robot manipulation is proposed. We adopt pivoting manipulation for the humanoid robot to move a bulky object without lifting in a stable and dexterous manner. In order to carry the object to a desired place, the humanoid should sometimes move through narrow areas surrounded by obstacles. We propose a roadmap multiplexing planning to allow the robot to leave the object near narrow places and to regrasp it from another position to continue carrying. We utilize visibility probabilistic roadmap (PRM) method as a preprocessing to capture the critical configurations for regrasping. Then a diffusion method is employed to plan the overall manipulation path including regrasping. The proposed method is verified through planning simulation including whole-body motions.

Human Robot collaborative task achievement requires adapted tools and algorithms for both decision making and motion computation. The human presence as well as its behavior must be considered and actively monitored at the decisional level for the robot to produce synchronized and adapted behavior. Additionally, having a human within the robot range of action introduces security constraints as well as comfort considerations which must be taken into account at the motion planning and control level. This paper presents a robotic architecture adapted to human robot interaction and focuses on two tools: a human aware manipulation planner and a supervision system dedicated to collaborative task achievement.

This paper emphasizes on the capacity of a humanoid robot to perform tasks that are difficult for other types of robots. It deals with manipulation of bulky objects. Such tasks require complicated manipulations involving the whole-body and fine coordination between legs, arms and torso motions. We introduce here a whole-body motion planner that allows a humanoid robot to autonomously plan a pivoting strategy that accounts for the various constraints: collision avoidance, legs-arms coordination and stability control. Based on a previous result by the authors [1] proving the small-time controllability of a pivoting system, the planner is proven to inherit from the probabilistic completeness of the samplingbased motion planning method it is built on. The geometric and kinematic capacity of the proposed planner is mainly demonstrated through simulations and experiments.

In military missions in hostile environments involving teams of UAVs flying in formation, it is important to get the maximum benefits of the auto-protection systems of each aircraft to enhance the global security and efficiency of the team. One way to achieve this is to select a proper configuration for the formation. In this paper, we present an approach to autonomously adapt the configuration of a formation and we focus on its evaluation within a realistic framework where each UAV is simulated independently and communicate through a network.

This paper presents a software component, the plan database, which provides the needed services to define plans, execute them and more importantly adapt them during execution. This plan database handles fully dynamic plans (insertion and removal of tasks), defines task transformation operators and provides tools for safe concurrent execution and modification of plans. These features are essential in multirobot and human-robot contexts, where tasks need to be easily passed between systems and plan adaptation helps coping with the unpredictability inherent to systems where multiple agent make decisions.

Robots' interaction with humans raises new issues for geometrical reasoning where the humans must be taken explicitly into account. We claim that a human-aware motion system must not only elaborate safe robot motions, but also synthesize good, socially acceptable and legible movement. This paper focuses on a manipulation planner and a placement mechanism that take explicitly into account its human partners by reasoning about their accessibility, their vision field and their preferences. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in presence or in synergy with humans.

Pivoting manipulation has such advantages as dexterity and safety over other methods to move bulky or heavy objects. In this paper we aim to show that a polyhedral object can be displaced to arbitrary position and orientation on a plane (i.e. such a pivoting system is controllable). More than that we show it is small time controllable, i.e. the reachable space from a starting point contains always a neighbor no matter how cluttered the environment is. As a consequence of this analysis, we propose a steering method to plan a manipulation path to be performed by a humanoid robot: first we use a classical nonholonomic path planner that accounts for the robot motion constraints, and then we transform that path into a sequence of pivoting operations. While the feasibility of elementary pivoting tasks has been already experienced by the humanoid robot HRP-2, we present here the very first simulations of the plans generated by our steering method.

Human-robot collaboration requires both communicative and decision making skills of a robot. To enable flexible coordination and turn-taking between human users and a robot in joint tasks, the robot's dialog and decision making mechanism have to be synchronized in a meaningful way. In this paper, we propose a integration framework to combine the dialog and the decision making processes. With this framework, we investigate various task negotiation situations for a social robot in a fetch-and-carry scenario. For the technical realization of the framework, the interface specification between the dialog and the decision making systems is also presented. Further, we discuss several challenging issues identified in our integration effort that should be adddressed in the future.

Human-robot collaborative task achievement requires the robot to reason not only about its current beliefs but also about the ones of its human partner. In this paper, we introduce a framework to manage shared knowledge for a robotic system dedicated to interactive task achievement with a human. In a first part, we define which beliefs should be taken into account ; we then explain a manner to achieve them using communication schemes. Several examples are presented to illustrate the purpose of beliefs management including a real experiment demonstrating a "give object" task between the Jido robotic platform and a human.

This paper addresses high-level robot planning issues for an interactive cognitive robot that has to act in presence or in collaboration with a human partner. We describe a task planner called HATP (for human aware task planner). HATP is especially designed to handle a set of human-centered constraints in order to provide "socially acceptable" plans that are oriented toward collaborative task achievement. We provide an overall description of HATP and discuss its main structure and algorithmic features.

Robot navigation in the presence of humans raises new issues for motion planning and control when the humans must be taken explicitly into account. We claim that a human aware motion planner (HAMP) must not only provide safe robot paths, but also synthesize good, socially acceptable and legible paths. This paper focuses on a motion planner that takes explicitly into account its human partners by reasoning about their accessibility, their vision field and their preferences in terms of relative human-robot placement and motions in realistic environments. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in the presence or in synergy with humans.

This paper presents the combined results of two studies that investigated how a robot should best approach and place itself relative to a seated human subject. Two live Human Robot Interaction (HRI) trials were performed involving a robot fetching an object that the human had requested, using different approach directions. Results of the trials indicated that most subjects disliked a frontal approach, except for a small minority of females, and most subjects preferred to be approached from either the left or right side, with a small overall preference for a right approach by the robot. Handedness and occupation were not related to these preferences. We discuss the results of the user studies in the context of developing a path planning system for a mobile robot.

International audience

The presence of humans should be explicitly taken into account in all steps of robot's design and particularly for robot motion. The robot should reason about human partner's accessibility, his vision field and potential shared motions and behave as a social being by respecting social rules and protocols. This paper describes the algorithms and results of a navigation planner that takes into account the human presence explicitly. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in presence and/or in synergy with human

Rackham is an interactive robot-guide that has been used in several places and exhibitions. This paper presents its design and reports on results that have been obtained after its deployment in a permanent exhibition. The project is conducted so as to incrementally enhance the robot functional and decisional capabilities based on the observation of the interaction between the public and the robot. Besides robustness and efficiency in the robot navigation abilities in a dynamic environment, our focus was to develop and test a methodology to integrate human-robot interaction abilities in a systematic way. We first present the robot and some of its key design issues. Then, we discuss a number of lessons that we have drawn from its use in interaction with the public and how that will serve to refine our design choices and to enhance robot efficiency and acceptability

The presence of humans in the robot environment brings new challenges to the robotic research. From low level functions to high level planners, clearly the human has to be taken into account in all the layers of the robot control system. Indeed, the robot has to behave socially in order to interact friendly with its human partners. This paper describes the development of several components (human detection and tracking, planning and supervision) that take into account humans explicitly with some preliminary results of their integration.

The future personal robot must be capable to work and take decisions in a dynamic human environment. One important capability of such a robot is manipulation. Grasp, as the beginning of any manipulation task is a key point. In this paper we present a grasp planner for manipulating real objects modeled by polyhedra. The algorithm is based on a random generation of grasp oriented by mass and inertial properties of the object. Recent techniques to filter and evaluate the quality of grasp are discussed.

This paper presents a decisional architecture and the associated algorithms for multi-UAV (unmanned aerial vehicle) systems. The architecture enables different schemes of decision distribution in the system, depending on the available decision making capabilities of the UAVs and on the operational constraints related to the tasks to achieve. The paper mainly focuses on the deliberative layer of the UAVs: we detail a planning scheme where a symbolic planner relies on refinement tools that exploit UAVs and environment models. Integration effort related to decisional features is highlighted, and preliminary simulation results are provided.

This paper deals with the task allocation problem in multi-robot systems. We propose a completely distributed architecture, where robots dynamically allocate their tasks while they are building their plans. We first focus on the problem of simple "goto" tasks allocation: our approach involves an incremental task allocation algorithm based on the Contract-Net protocol. We introduce a parameter called equity coefficient in order to equilibrate the workload between the different robots and to control the triggering of the auction process. Then, we address the problem raised by temporal constraints between tasks by dynamically specifying temporary hierarchies among the tasks. Tests run in simulation quantify the benefits of our improvements.

We propose a resolution scheme that is aimed to solve a wide range of multi-robot planning problems in a 3D geometrical environment. This complements our efforts in developing manipulation planning algorithms to deal with multiple robots and several movable objects problems. While the elementary planning step relies on Probabilistic Roadmap Methods (PRMs), the main contribution here is a reasoning level that adapts its control over the construction and extension of a number of roadmaps. The coherence between roadmaps and the hierarchical search process are done through a new type of graph built for a subset of objects or robots in the environment, called the Elementary Kinematic Graph. This paper describes the main ingredients of the proposed framework, and its first results.

A methodology for computing the maximum velocity profile for a planned trajectory of the robot is described in this paper. The profile is computed considering the robot and environment dynamics as well as the constraints of the sensing apparatus. The mobile objects can be arbitrary in number and their direction and velocity of motion is not known. The only known information about the moving objects is the maximum velocity they can possess. The robot that moves with the computed velocity profile can assure from its side that it would not collide onto any of the numerous moving objects that could intercept its future trajectory. The methodology has been incorporated onto a motion planner for a nonholonomous robot and the results presented. The motivation here is to facilitate the process of having safe and understanding robots. Hence the planned velocity profiles are in general conservative though the robot could perhaps do better on-line. However at planning time the robot's immobility before collision is guaranteed.

We propose in this paper a resolution scheme that is aimed to be relevant for a large class of manipulation planning problems. This endeavor complements our efforts in developing manipulation planning algorithms. Indeed, we are convinced that a higher level of problems complexity, and particularly those involving multiple robots and multiple objects, will be accessible thanks to the introduction of a symbolic reasoning level. The resolution scheme relies on probabilistic roadmap methods (PRMs) and on a reasoning level that adaptively controls the construction and extension of a number of roadmaps. We consider this symbolic level as a step towards a systematic approach to integrate task planning and geometric planning in better conditions than through a gross, and somewhat, artificial hierarchical decomposition. This paper describes the main ingredients of the proposed framework, and its first results.

Autonomous robots are complex machines embedding: software modules implementing basic functions such as servo loops to follow a path, or to compute stereo correlation of a pair of images; software to take into account the interactions between the robot components; programs which deal with the supervision and the planning of the mission. These components are integrated within an architecture which defines an organization and a methodology for robot operation.

This paper reports on our efforts to extend the plan-merging paradigm, in order to improve its ability to adapt to various execution contexts. They concern the algorithmic as well as the architectural issues. The scheme can be parameterized, allowing to choose and even to update online robot priorities, and to take into account various constraints imposed by the application. These lead to different coordination policies. The architectural improvements allowed a better distribution of the computation load, leading to more complex applications, for instance, the coordination of multiple mobile manipulators. The result is a more flexible scheme that should widen its application fields. We describe the main ingredients of this new scheme and illustrate its implementation through two examples.

In the stream of research that aims to speed up practical planners, we propose an approach to task planning based on probabilistic roadmap methods (PRM). Our contribution is twofold. The first issue concerns the development of ShaPer, a task planner that is able to deal efficiently with large problems. ShaPer "captures" the structure of the task space. The second contribution involves promising results on robot task planning. This is obtained through an analysis of the task space structure that exhibits the relation between task and geometric reasoning for a given robot task. To illustrate such an approach, we solve a complex problem where motion and task planning are closely interleaved.

Presents the global nearness diagram navigation system for mobile robots. The GND generates motion commands to drive a robot safely between locations, whilst avoiding collisions. This system has all the advantages of using the reactive scheme nearness diagram (ND), while having the ability to reason and plan globally (reaching global convergence to the navigation problem). This framework has been extensively tested using a holonomic mobile base equipped with a laser range-finder. Experiments in unknown, unstructured, dynamic and complex environments are reported to validate the system.

In mobile robotics, pure geometric representations may not be well suited for navigation in large scale environments. New models combine topological and metrical information to give compact and efficient representations. We briefly review the outlines of the construction of our Voronoi-like graph and give further details about its implementation in a real environment. Several trials made around our lab demonstrate the ability of our modeling scheme to recognize topological features of the environment and use them to detect loops and relocalize the robot position.

We propose a new approach to treat a class of cooperative issues in the multi-robot context. These issues are associated with the common use of some entities, called mechanisms. A mechanism can be seen as a generalization of the notion of resource. The robots can modify its state directly or through requests. The robots can also share its utilization. Multi-robot cooperation can be expressed as a distributed decisional process that tends to solve, detect and treat resource conflict situations as well as sources of inefficiency. We discuss these issues and illustrate them through a simulated system which allows a number of autonomous robots to plan and perform cooperatively a set of servicing tasks in a hospital environment.

The authors previously (1998) argued that the LAAS architecture is one of the most suitable for mobile robot control. This statement may seem over-optimistic, not to say pretentious and unverifiable. After all, can we compare architectures? can we set up benchmarks? or can we measure how good an architecture is compared to another? An architecture defines organization principles, integration methods and supporting tools. Comparing those tools, methods and principles may sometime end up in sterile controversies. However, we think there are means to measure the overall quality (or interest) of an architecture. Development time is for example one relevant criterion. Basically, using a specific architecture, how long does it take to integrate a complete demonstration, including nontrivial decisional capabilities, from the low level functional modules up to the supervisory level? This may seem a rather weak measure of architecture quality; however, it encompasses properties such as genericity and adaptability, ease of design and programming, extensibility and robustness. In this paper we describe our recent experience in integrating a complete demonstration from scratch in 40 days using the LAAS architecture.

Discusses a general architecture where various schemes for multi-robot task achievement can be integrated called "M+ Cooperative task achievement". The main originality comes from its ability to allow the robots to detect-in distributed and cooperative manner-resource conflict situations as well as sub-optimalities. Different decisions are performed by the robots such as actions re-scheduling, suppression of redundancies and opportunistic enhancement of the global performance. Finally, we illustrate its use through a simulated system, which allows a number of robots to plan and perform cooperatively a set of tasks in a hospital environment.

Presents our project to develop a personal robot called Diligent. We first discuss the main issues involved in personal robotics. Then we analyze some of the key features implied by what we call "human-friendly navigation". Finally, we present the current state of Diligent. Diligent is already able to navigate safely and repeatedly in a relatively large environment. It is also capable of some preliminary human-robot interactions through very simple and intuitive commands.

In the field of mobile robotics, one important issue is to allow the robot to navigate in an a priori unknown and non-specific large scale environment. Large dimensions raise strong limitations of geometric modeling, and topological or mixed metric-topological models are now studied to better fit the problem. We present a method for incrementally building a topological model of an indoor environment from sensor range data. The approach consists in merging each local perception of the topology with the current state of the global graph. This local topology is captured through the construction of a Voronoi-like graph that takes into account not only visible features but also visibility constraints (hidden regions, limited sensing ranges,...). We give the outline of the method and show first encouraging results on real data.

We present and discuss a distributed scheme for multi-robot cooperation. It integrates mission planning and task refinement as well as cooperative mechanisms adapted from the contract net protocol framework. We discuss its role and how it can be integrated as a component of a complete robot control system. We also discuss how it handles distributed task allocation and achievement as well as cooperative reaction to contingencies. Finally, we illustrate its use through a simulated system, which allows a number of robots to perform load transfer tasks in a route network environment.

We present and discuss the use of a generic scheme for multi-robot cooperation called the "plan-merging paradigm" for managing a large fleet of autonomous mobile robots. Each robot, autonomously and incrementally builds and executes its own plans taking into account the multi-robot context obtained by collecting the current plans and goals of the other robots. We describe the overall system architecture and discuss the properties of our cooperative scheme. We show how the plan-merging paradigm (PMP) can be used in a hierarchical manner and how it "fills the gap" between centralized planning and distributed execution. We finally illustrate this scheme through an implemented system which allows a fleet of autonomous mobile robots to perform load transfer tasks in a route network environment. The central activity is limited to task allocation and important gains are obtained in system flexibility and robustness to execution contingencies. Simulations (using up to 30 robots) as well as experiments with real robots (3) are presented and discussed.

Our approach in proposing the plan-merging paradigm as a cooperation scheme was to allow an efficient distribution of the decisions for a better reactivity to contingencies for multi-robot applications with loosely coupled tasks. The paradigm proved to be quite efficient because it exploits the fact that most conflicts can be solved locally and because it allows a finer overlapping between plan refinement, plan coordination and execution. We develop in this paper the mechanisms which allow one to extend this paradigm in order to obtain a scheme which distributes as much as possible the decision processes for planning and coordination while maintaining two key features: the coherence of the global system and the ability to detect the situations where it is not applicable; and a localized management of the planning and coordination processes with, in particularly intricate situations, a progressive transition to more global schemes which may "degrade" to a unique and centralized planning activity.

We discuss Procedural Reasoning System (PRS) as a high-level control and supervision language adapted to autonomous robots to represent and execute procedures, scripts and plans in dynamic environments. We discuss the main reasons why PRS is well suited for this type of application: (1) The semantics of its plan (procedure) representation, which is important for plan execution and goal refinement; (2) Its ability to construct and act on partial (rather than complete) plans; (3) Its ability to pursue goal-directed tasks while at the same time being responsive to changing patterns of events in bounded time; (4) Its facilities for managing multiple tasks in real-time; (5) Its default mechanisms for handling stringent real-time demands of its environment; and (6) Its meta-level (or reflective) reasoning capabilities. C-PRS has been used to implement an embedded control and supervision system for autonomous mobile robots in two different experimentations that we briefly present. We conclude with some suggestions to further develop C-PRS and with a short review of related work.

In the MARTHA project, a large number of robots in a harbour are given the global task of transporting containers from one area to another. The global decision-making process of allocating robots to those predefined tasks can be viewed as a scheduling and resource allocation problem, which is addressed here in a centralised way. Imprecision of temporal constraints makes it meaningless to search for a strict optimal schedule. The authors' approach interleaves task allocation and execution, scheduling in a sliding short-term horizon, as the execution process runs, and providing near-optimal solutions. For large applications the complexity of temporal management is a crucial issue. The authors present a graph decomposition technique, leading to nearly-constant time temporal propagation, without any loss of information.

We present and discuss a generic cooperative scheme for multirobot cooperation. It is based on an incremental and distributed plan-merging process. We discuss the properties of this cooperative scheme (coherence, detection of dead-lock situations) as well as the class of applications for which it is well suited. We also discuss how this paradigm "fills the gap" between centralized planning and distributed execution. We show how this scheme can be used in a hierarchical manner, and in contexts where planning is performed in parallel with plan execution. We finally illustrate it through an implemented system which allows a fleet of more than ten autonomous mobile robots to perform load transfer tasks in a route network environment with a very limited centralized activity and important gains in system flexibility and robustness to execution contingencies.

This paper presents an approach we have recently developed for multi-robot cooperation. It is based on a paradigm where robots incrementally merge their plans into a set of already coordinated plans. This is done through exchange of information about their current state and their future actions. This leads to a generic framework which can be applied to a variety of tasks and applications. The paradigm, called plan-merging paradigm, is presented and illustrated through its application to planning, execution and control of a large fleet of a autonomous mobile robots for load transport tasks in a structured environment.

This paper addresses the problem of planning the motions of a circular mobile robot moving amidst polygonal obstacles with uncertainty in robot control and sensing. The robot is equipped with sensors which, if properly used, may provide information to overcome the uncertainty accumulated during the motions. The position sensor is based on dead-reckoning, the error then results in a cumulative uncertainty. A proximity sensor may be used to localize the robot with respect to the obstacles of the environment. The robot can also gain information by entering inside landmark areas where the position error is assumed to be bounded. The authors describe a planner which produces robust motion strategies composed of sensor-based motion commands which guarantee that, given an explicit model of the error accumulated by the motion commands, the robot can reach safely its goal with an error lower than a pre-specified value. It is based on a propagation of a numerical potential and on a geometric analysis of the reachability of environmental features. This planner exhibits a set of powerful capabilities: while it allows motion primitives which accumulate uncertainty to be considered, it is able, whenever possible, to navigate without relocalizing the robot when the task does not impose it, and also to make a proper use of the sensors. Several examples run with the planner are presented.

This paper presents an implemented system which allows one to run a fleet of autonomous mobile robots in a route network with a very limited centralized activity. The robots are provided with all the necessary ingredients for planning and executing navigation missions in a multi-robot context. Multi-robot cooperation is based on a generic paradigm called plan-merging paradigm, where robots incrementally merge their plans into a set of already coordinated plans. The robot architecture is derived from the generic architecture developed at LAAS. A 3D graphic environment system allows one to display a complete system composed of a dozen (or more) robots, each running on an independent workstation. An example is presented together with numerical results on the system behavior.

This paper reports on a recent work we have conducted concerning the development and the implementation of a robust motion planner for a mobile robot in a polygonal environment and in presence of uncertainty in robot control and sensing. Such a planner takes explicitly into account the uncertainty in robot control and produces a robust motion plan composed of sensor-based motion commands. The main originality of this approach is that it is able to account for a set of motion commands which may accumulate errors. It is based on a geometric analysis of the reachability and it generates motion strategies which may allow the robot to reduce its uncertainty.>

Considers the problem of programming a remote intervention robot with a high degree of autonomy and reasoning abilities. The authors propose a control strategy, called task-level teleprogramming, based on a specific architecture. After a brief presentation of their approach, the authors discuss in this paper developments, mainly a teleprogramming language, and illustrate them with an example.>

The authors consider the class of intervention robots that have to perform tasks in remote environments which may impose painful, harsh or utterly dangerous conditions to humans. They informally characterize this class of problems and briefly discuss several basic concepts which underlie the current approaches. The remote autonomous robot concept that they have developed fulfills the requirements deduced of these basic ideas. The authors illustrate the real world possibilities of the concept by describing the features of two application projects (AMR, VAP) that they believe to be significant and demonstrative. The last section is devoted to the description of a generic experiment carried on with ADAM, an all-terrain full size rover. It includes the system architecture implementation and some of the experimental results.

The authors describe an implemented integrated system allowing a mobile robot to plan its actions, taking into account temporal constraints, and to control their execution in real time. The general architecture has three levels, and the approach is related to hierarchical planning: the plan produced by the temporal planner is further refined at the control level, which in turn supervises its execution by a functional level. The framework of the French Mars Rover project VAP is used as an illustration of the various aspects discussed.>

The authors address the motion planning problem for a robot and a movable object within polygonal obstacles. Motion planning in this context appears as a constrained instance of the coordinated motion planning problem for two robots. The object cannot move itself, it only does when the robot grasps it. A topological property that characterizes the existence of solutions in the subspace of configurations where the robot touches the object is discussed. This property provides a general resolution scheme which is applied to the special case of a convex polygonal robot moving in translation with a convex polygonal object amidst polygonal obstacles. The algorithm has been implemented, and experimental results are discussed.>

We consider the class of intervention mobile robots that huve to perform tasks in ill-known environments which are often remote or of difflicult or dangerous access. An outstanding example of this would be a Mars Rover designed to carry out planet exploration, scientific measurements and experiments, according lo the requeslis of Earth based scientists over a long period of time, e.g. one Martian year. The lack of structure in the environment of this class of intervention robots, as well as the nonrepetitive nature of the tasks, clearly require eficient task programming and constant adaptation to the environment by the robot. We contend that this class of problems demands a specific decisional architecture which couples “intelligence” at the programming level with “intelligence” at the level of the physical machine. This second requirement implies that the machine exhibits the attributes of an autonomous intelligent robot. In this paper we develop the concepts of a generic architecture aimed to achieve task control of a remote intervention robot. The robot is fully autonomous at task level. It receives tasks assignements that it transforms into sequences of actions using ats own interpretation and planning capacities, and executes these actions while being ,reactive to asynchronous events and environment conditions. We have developed the approach presented here in the context of several domain-oriented projects, and fully implemented a first version of the concepts in our newest mobile robot, HILARE II. The framework of the french Mars Rover project VAP is used as an illustration of some crucial aspects discussed in the paper.

Given a workcell program defined as a set of repetitive tasks to be concurrently performed, one can associate with each workcell element (robots, vision systems, etc.) a repetitive partial order of operations; together, they define a workcell cycle. An evaluation of such cycles in terms of cycle time, task throughput, and machine utilization is proposed. To automate the sequencing analysis, a Prolog-based decision-support system, SAGE (sequence analysis by generalized enumeration), is presented. Given the specification of the work-cell program in terms of the operations to be performed by each workcell element, SAGE constructs all feasible cycles for exploring the space for all possible time evolutions of workcell behavior. Three variations on the same multirobot assembly example are used to illustrate the approach.>

The automatic failure recovery of assembly tasks in robot workcells is discussed. An efficient scheme for the design of autonomous workcell system management systems is presented. In this approach an online management system performs two functions: task scheduling and task execution control (the latter embeds failure recovery). Task execution control is decomposed into three functions: monitoring of tasks, failure analysis, and recovery. Methods of designing the modules that perform these functions are presented. Failure detection is performed by a sensor-based monitoring module, failure diagnosis is performed by a rule-based failure analysis module, and failure recovery is accomplished by a two-level recovery module.>

The authors propose a representation of positioning uncertainty for robot manipulation tasks and a method by which it can be propagated through robot programs. The objective is to integrate error propagation and analysis into a task-level programming system in order to produce robust programs. This integration is discussed with reference to an example. The method consists of evaluating progressively the uncertainty through a sequence of actions and maintaining a symbolic trace of computations. When a constraint is violated, the system determines the predominant source of uncertainty in order to guide the planning process.>

Performing complex assembly tasks requires decisional capabilities available on-line, in order to handle various events (sensor conditions, part arrivals, failure detection...), that occur during execution. In this paper, we describe NNS, a complete on-line system for multi-robot assembly workcells, that deals with problems of execution monitoring, action scheduling, and failure diagnosis and recovery. NNS is distributed in different decisional layers with their own reasoning methods and state representation; it uses an "execution model" that embeds various knowledge bases built off-line. This approach fits well with requirements of flexibility and efficiency.

This paper is divided into four sections. The first section discusses Flexible Assembly Cell (FAC) programming requirements. Sections 2 and 3 present an approach to FAC modelling and programming. The intended objective is to build a programming and decisional environment which allows construction of assembly applications, and to endow the on-line system with decision-making capabilities. The last section describes the current state of an actual system which is a first approach to meeting the discussed requirements.

This paper addresses issues of integration and programming of complex robotics systems. It describes the underlying approach, the design and implementation of the NNS system, an interactive LISP-based environment for the integration and setting up of Flexible Assembly Cells. Programming, debugging, parallelism managing and interaction with decision processes are carried out in the same environment. NNS provides a suitable framework for the design of highly adaptative robotics systems and their use in the experiment of complex manipulation tasks.