This paper proposes a knowledge-driven teleoperation framework that enables multiple operators to command a team of robots to execute complex tasks in an efficient and intuitive manner. The framework leverages a shared knowledge base that captures domain-specific information and procedural knowledge about the task at hand. This knowledge base is used by a hybrid planner to generate context-specifically relevant commands for supervised autonomy robot command as well as direct teleoperation modes. By filtering the available commands, the operators are guided in their decision-making towards efficient task completion. This paper further extends our knowledge driven approach to address the switching between multiple operators and robotic assets, with the aim to be able scale up human-robot team for space exploration. Overall, this work represents a step towards more intelligent and collaborative teleoperation systems. The described system will be used in the Surface Avatar ISS-to-ground experiments slated for July 2023.

From robotic space assistance to healthcare robotics, there is increasing interest in robots that offer adaptable levels of autonomy. In this paper, we propose an action representation and planning framework that is able to generate plans that can be executed with both shared control and supervised autonomy, even switching between them during task execution. The action representation - Constraint Action Templates (CATs) - combine the advantages of Action Templates [1] and Shared Control Templates [2]. We demonstrate that CATs enable our planning framework to generate goal-directed plans for variations of a typical task of daily living, and that users can execute them on the wheelchair-robot EDAN in shared control or in autonomous mode.

Nowadays, robots are mechanically able to perform highly demanding tasks, where AI-based planning methods are used to schedule a sequence of actions that result in the desired effect. However, it is not always possible to know the exact outcome of an action in advance, as failure situations may occur at any time. To enhance failure tolerance, we propose to predict the effects of robot actions by augmenting collected experience with semantic knowledge and leveraging realistic physics simulations. That is, we consider semantic similarity of actions in order to predict outcome probabilities for previously unknown tasks. Furthermore, physical simulation is used to gather simulated experience that makes the approach robust even in extreme cases. We show how this concept is used to predict action success probabilities and how this information can be exploited throughout future planning trials. The concept is evaluated in a series of real world experiments conducted with the humanoid robot Rollin' Justin.

Light-weight robotic manipulators can be used to restore the manipulation capability of people with a motor disability. However, manipulating the environment poses a complex task, especially when the control interface is of low bandwidth, as may be the case for users with impairments. Therefore, we propose a constraint-based shared control scheme to define skills which provide support during task execution. This is achieved by representing a skill as a sequence of states, with specific user command mappings and different sets of constraints being applied in each state. New skills are defined by combining different types of constraints and conditions for state transitions, in a human-readable format. We demonstrate its versatility in a pilot experiment with three activities of daily living. Results show that even complex, high-dimensional tasks can be performed with a low-dimensional interface using our shared control approach.

Injuries, accidents, strokes, and other diseases can significantly degrade the capabilities to perform even the most simple activities in daily life. A large share of these cases involves neuromuscular diseases, which lead to severely reduced muscle function. However, even though affected people are no longer able to move their limbs, residual muscle function can still be existent. Previous work has shown that this residual muscular activity can suffice to apply an EMG-based user interface. In this paper, we introduce DLR's robotic wheelchair EDAN (EMG-controlled Daily Assistant), which is equipped with a torque-controlled, eight degree-of-freedom light-weight arm and a dexterous, five-fingered robotic hand. Using electromyography, muscular activity of the user is measured, processed and utilized to control both the wheelchair and the robotic manipulator. This EMG-based interface is enhanced with shared control functionality to allow for efficient and safe physical interaction with the environment.

Intelligent robotic coworkers are considered a valuable addition in many application areas. This applies not only to terrestrial domains, but also to the exploration of our solar system. As humankind moves toward an ever increasing presence in space, infrastructure has to be constructed and maintained on distant planets such as Mars. AI-enabled robots will play a major role in this scenario. The space agencies envisage robotic co-workers to be deployed to set-up habitats, energy, and return vessels for future human scientists. By leveraging AI planning methods, this vision has already become one step closer to reality. In the METERON SUPVIS Justin experiment, the intelligent robotic coworker Rollin’ Justin was controlled from Astronauts aboard the International Space Station (ISS) in order to maintain a Martian mock-up solar panel farm located on Earth to demonstrate the technology readiness of the developed methods. For this work, the system is demonstrated at AAAI 2019, controlling Rollin’ Justin located in Munich, Germany from Honolulu, Hawaii.

Human teleoperation of robots and autonomous operations go hand in hand in today's service robots. While robot teleoperation is typically performed on low to medium levels of abstraction, automated planning has to take place on a higher abstraction level, i.e. by means of semantic reasoning. Accordingly, an abstract state of the world has to be maintained in order to enable an operator to switch seamlessly between both operational modes. We propose a novel approach that combines simulation based geometric tracking and semantic state inference by means of so called State Inference Entities to overcome this issue. We also demonstrate how Evolutionary Strategies can be employed to refine simulation parameters. All experiments are demonstrated in real-world experiments conducted with the humanoid robot Rollin' Justin.

Many household chores and industrial manufacturing tasks require a certain compliant behavior to make deliberate physical contact with the environment. This compliant behavior can be implemented by modern robotic manipulators. However, in order to plan the task execution, a robot requires generic process models of these tasks which can be adapted to different domains and varying environmental conditions. In this work we propose a classification of compliant manipulation tasks meeting these requirements, to derive related actions for automated planning. We also present a classification for the sub-category of wiping tasks, which are most common and of great importance in service robotics.We categorize actions from an object-centric perspective to make them independent of any specific robot kinematics. The aim of the proposed taxonomy is to guide robotic programmers to develop generic actions for any kind of robotic systems in arbitrary domains.

Many houseworks such as cleaning the floor or wiping the windows require to manipulate tools over wide areas. It is necessary to move along a path while manipulating a tool with the whole body and applying exactly the right amount of force to successfully accomplish the task. So mastering such a challenge demands detailed knowledge about the involved objects and the underlying process models. Reasoning about an appropriate parameterization of the task is thereby essential. In this paper we propose a combination of object-centered hybrid reasoning and compliant force control to solve complex whole-body mobile manipulation issues. Depending on the objects involved in the task, an appropriate controller is selected and automatically parameterized. The methods are validated in an elaborate experiment on the humanoid robot Rollin' Justin.

This paper presents an approach for computing power grasps for hands with kinematic structure similar to the human hand, which allows the implementation of strategies inspired in human grasping actions. The proposed method first samples the object surface to look for the best spots for creating an opposing grasp with two or three fingers, and then aligns the other fingers to match the local curvature of the object surface. Different grasp strategies are considered, depending on the relative size of the object with respect to the hand, and on the location of potential obstacles in the environment. Several application examples are provided with two different hand models.

In autonomous bimanual operation of a robot, parallelized planning and execution of a task is essential. Elements of a task have different functional and spatial relationships. They may depend on each other and have to be executed in a specific order or they may be independent and their order can be determined freely. Consequently, individual actions can be planned and executed in parallel or not. In a proof of concept, this paper shows that the structure of a task and its mapping onto subordinate planners can significantly influence planning speed and task execution. Independent tasks are planned using two parallel path planners. Dependent tasks are planned using one path planner for both arms. Using a simple, yet expandable experimentation scenario, the resulting recommendations for parameterizing path planners are verified on a humanoid robot. For execution on the real robot a violation of the rigid body model used in path planners had to be addressed.