This paper proposes a knowledge-driven teleoperation framework that enables multiple operators to command a team of robots to execute complex tasks in an efficient and intuitive manner. The framework leverages a shared knowledge base that captures domain-specific information and procedural knowledge about the task at hand. This knowledge base is used by a hybrid planner to generate context-specifically relevant commands for supervised autonomy robot command as well as direct teleoperation modes. By filtering the available commands, the operators are guided in their decision-making towards efficient task completion. This paper further extends our knowledge driven approach to address the switching between multiple operators and robotic assets, with the aim to be able scale up human-robot team for space exploration. Overall, this work represents a step towards more intelligent and collaborative teleoperation systems. The described system will be used in the Surface Avatar ISS-to-ground experiments slated for July 2023.

Nowadays, robots are mechanically able to perform highly demanding tasks, where AI-based planning methods are used to schedule a sequence of actions that result in the desired effect. However, it is not always possible to know the exact outcome of an action in advance, as failure situations may occur at any time. To enhance failure tolerance, we propose to predict the effects of robot actions by augmenting collected experience with semantic knowledge and leveraging realistic physics simulations. That is, we consider semantic similarity of actions in order to predict outcome probabilities for previously unknown tasks. Furthermore, physical simulation is used to gather simulated experience that makes the approach robust even in extreme cases. We show how this concept is used to predict action success probabilities and how this information can be exploited throughout future planning trials. The concept is evaluated in a series of real world experiments conducted with the humanoid robot Rollin' Justin.

Intelligent robotic coworkers are considered a valuable addition in many application areas. This applies not only to terrestrial domains, but also to the exploration of our solar system. As humankind moves toward an ever increasing presence in space, infrastructure has to be constructed and maintained on distant planets such as Mars. AI-enabled robots will play a major role in this scenario. The space agencies envisage robotic co-workers to be deployed to set-up habitats, energy, and return vessels for future human scientists. By leveraging AI planning methods, this vision has already become one step closer to reality. In the METERON SUPVIS Justin experiment, the intelligent robotic coworker Rollin’ Justin was controlled from Astronauts aboard the International Space Station (ISS) in order to maintain a Martian mock-up solar panel farm located on Earth to demonstrate the technology readiness of the developed methods. For this work, the system is demonstrated at AAAI 2019, controlling Rollin’ Justin located in Munich, Germany from Honolulu, Hawaii.

Human teleoperation of robots and autonomous operations go hand in hand in today's service robots. While robot teleoperation is typically performed on low to medium levels of abstraction, automated planning has to take place on a higher abstraction level, i.e. by means of semantic reasoning. Accordingly, an abstract state of the world has to be maintained in order to enable an operator to switch seamlessly between both operational modes. We propose a novel approach that combines simulation based geometric tracking and semantic state inference by means of so called State Inference Entities to overcome this issue. We also demonstrate how Evolutionary Strategies can be employed to refine simulation parameters. All experiments are demonstrated in real-world experiments conducted with the humanoid robot Rollin' Justin.