Over the last years, powerful methods for solving specific perception problems such as object detection, pose estimation or scene understanding have been developed. While performing mobile manipulation actions, a robot’s perception framework needs to execute a series of these methods in a specific sequence each time it receives a new perception task. Generating proficient combinations of vision methods to solve individual perception tasks remains a challenge, as the combination depends on the requirements of the task and the capabilities of the robot’s hardware.In this paper, we propose RoboKudo, an open-source knowledge-enabled perception framework that leverages the strengths of the Unstructured Information Management (UIM) principle and the flexibility of Behavior Trees to model task-specific perception processes. The framework can combine state-of-the-art computer vision methods to satisfy the requirements of each perception task and scales to different robot platforms. The generality and effectiveness of the framework are evaluated in real world experiments where it solves various perception tasks in the context of mobile manipulation actions in a household domain. Code and additional material are available at https://robokudo.ai.uni-bremen.de/rkop.

In this paper, we present an experiment, designed to investigate and evaluate the scalability and the robustness aspects of mobile manipulation. The experiment involves performing variations of mobile pick and place actions and opening/closing environment containers in a human household. The robot is expected to act completely autonomously for extended periods of time. We discuss the scientific challenges raised by the experiment as well as present our robotic system that can address these challenges and successfully perform all the tasks of the experiment. We present empirical results and the lessons learned as well as discuss where we hit limitations.

In this paper, we present an approach and an implemented framework for applying plan transformations to real-world mobile manipulation plans, in order to specialize them to the specific situation at hand. The framework can improve execution cost and achieve better performance by autonomously transforming robot's behavior at runtime. To demonstrate the feasibility of our approach, we apply three example transformations to the plan of a PR2 robot performing simple table setting and cleaning tasks in the real world. Based on a large amount of experiments in a fast plan projection simulator, we make conclusions on improved execution performance.

In this paper, we present an approach and an implemented pipeline for transferring data acquired from observing humans in virtual environments onto robots acting in the real world, and adapting the data accordingly to achieve successful task execution. We demonstrate our pipeline by inferring seven different symbolic and subsymbolic motion parameters of mobile pick and place actions, which allows the robot to set a simple breakfast table. We propose an approach to learn general motion parameter models and discuss, which parameters can be learned at which abstraction level.

Plan execution on real robots in realistic environments is underdetermined and often leads to failures. The choice of action parameterization is crucial for task success. In this paper, we present a mechanism for a robot that is acting in a real-world environment to think ahead of time with fast plan projection and, thereby, choose action parameterizations that are predicted to lead to successful execution. For finding causal relationships between action parameterizations and task success, we provide the robot with means for plan introspection and propose a systematic and hierarchical plan structure to support that. We evaluate our approach by showing how a PR2 robot, when equipped with the proposed system, is able to choose action parameterizations that increase task execution success rates and overall performance of fetch and place actions in a real world setting.

To enable robots to perform human-level tasks flexibly in varying conditions, we need a mechanism that allows them to exchange knowledge between themselves for crowd-sourcing the knowledge gap problem. One approach to achieve this is to equip a cloud application with a range of encyclopedic knowledge (i.e. ontologies) and execution logs of different robots performing the same tasks in different environments. In this paper, we show how knowledge exchange between robots can be done using OPENEASE as the cloud application. We equipped OPENEASE with ontologies about the kitchen domain, execution logs of three robots operating in two different kitchens, and semantic descriptions of both environments. By addressing two different use cases, we show that two PR2 robots and one Fetch robot can successfully adapt each other's plan parameters and sub symbolic data to the experiments that they are conducting.

With the advancements in robotic technology and the progress in human-robot interaction research, the interest in deploying mixed human-robot teams in rescue missions is increasing. Due to their complementary capabilities in terms of locomotion, visibility and reachability of areas, human-robot teams are considerably deployed in real-world settings, albeit the robotic agents in such scenarios are normally fully teleoperated. A major barrier to successful and efficient mission execution in those teams is the lack of cognitive skills in robotic systems. In this paper, we present a cognition-enabled framework and an implemented system where robotic agents are equipped with cognitive capabilities to naturally communicate with humans and autonomously perform tasks. The framework allows for natural tasking of robots, reasoning about robot behavior, capabilities and actions, and a common belief state representation for shared mission awareness of robots and human operators.

This paper tackles the problem of generalizing robot control programs over multiple objects, tasks and environments, based on the concept of action descriptions. These are abstract, general, semantic descriptions of an action that are augmented during execution with subsymbolic parameters specific to the context at hand. The parameters are inferred through reasoning rules, which extract the context from the action description and the belief state of the robot. The proposed system scales well with increasing number of reasoning rules required to support the knowledge-intensive manipulation tasks. The architecture combines the high-level robot control program with the reasoning engine in a modular way, thus improving the scalability of the system. The approach is validated in the context of setting a table with a PR2 robot.