In this paper we discuss how the combination of modern technologies in “big data” storage and management, knowledge representation and processing, cloud-based computation, and web technology can help the robotics community to establish and strengthen an open research discipline. We describe how we made the demonstrator of a EU project review openly available to the research community. Specifically, we recorded episodic memories with rich semantic annotations during a pizza preparation experiment in autonomous robot manipulation. Afterwards, we released them as an open knowledge base using the cloud- and web-based robot knowledge service OPENEASE. We discuss several ways on how this open data can be used to validate our experimental reports and to tackle novel challenging research problems.

Making future autonomous robots capable of accomplishing human-scale manipulation tasks requires us to equip them with knowledge and reasoning mechanisms. We propose Open-EASE, a remote knowledge representation and processing service that aims at facilitating these capabilities. Open-EASE gives its users unprecedented access to the knowledge of leading-edge autonomous robotic agents. It also provides the representational infrastructure to make inhomogeneous experience data from robots and human manipulation episodes semantically accessible, and is complemented by a suite of software tools that enable researchers and robots to interpret, analyze, visualize, and learn from the experience data. Using Open-EASE users can retrieve the memorized experiences of manipulation episodes and ask queries regarding to what the robot saw, reasoned, and did as well as how the robot did it, why, and what effects it caused.

The vision of the RoboEarth project is to design a knowledge-based system to provide web and cloud services that can transform a simple robot into an intelligent one. In this work, we describe the RoboEarth semantic mapping system. The semantic map is composed of: 1) an ontology to code the concepts and relations in maps and objects and 2) a SLAM map providing the scene geometry and the object locations with respect to the robot. We propose to ground the terminological knowledge in the robot perceptions by means of the SLAM map of objects. RoboEarth boosts mapping by providing: 1) a subdatabase of object models relevant for the task at hand, obtained by semantic reasoning, which improves recognition by reducing computation and the false positive rate; 2) the sharing of semantic maps between robots; and 3) software as a service to externalize in the cloud the more intensive mapping computations, while meeting the mandatory hard real time constraints of the robot. To demonstrate the RoboEarth cloud mapping system, we investigate two action recipes that embody semantic map building in a simple mobile robot. The first recipe enables semantic map building for a novel environment while exploiting available prior information about the environment. The second recipe searches for a novel object, with the efficiency boosted thanks to the reasoning on a semantically annotated map. Our experimental results demonstrate that, by using RoboEarth cloud services, a simple robot can reliably and efficiently build the semantic maps needed to perform its quotidian tasks. In addition, we show the synergetic relation of the SLAM map of objects that grounds the terminological knowledge coded in the ontology.

In many cases, the success of a manipulation action performed by a robot is determined by how it is executed and by how the robot moves during the action. Examples are tasks such as unscrewing a bolt, pouring liquids and flipping a pancake. This aspect is often abstracted away in AI planning and action languages that assume that an action is successful as long as all preconditions are fulfilled. In this paper we investigate how constraint-based motion representations used in robot control can be combined with a semantic knowledge base in order to let a robot reason about movements and to automatically generate executable motion descriptions that can be adapted to different robots, objects and tools.

We propose a method to learn the partially-ordered structure inherent in human everyday activities from observations by exploiting variability in the data. Using statistical relational learning, the system extracts a full-joint probability distribution over the actions that form a task, their (partial) ordering, and their properties. Relevant action properties and relations among actions are learned as those that are consistent among the observations. The models can be used for classifying action sequences, for determining which actions are relevant for a task, which objects are usually manipulated, and which action properties are typical for a person. We evaluate the approach on synthetic data sampled from partial-order trees as well as two real-world data sets of humans activities: the TUM kitchen data set and the CMU MMAC data set. The results show that our approach outperforms sequence-based models like Conditional Random Fields for classifying observations of activities that allow a large amount of variation.

In this paper, we discuss how networked robot architectures can facilitate the development, deployment, management and adaptation of distributed robotic applications. Our aim is to modularize applications by factoring out environment, task-, domain-, and robot-specific knowledge components and representing them explicitly in a formal knowledge base that is shared between the robots and service applications. Robot control decisions can then be formulated in terms of inference tasks that are evaluated based on this knowledge during task execution. The explicit and modular knowledge representation allows human operators with different areas of expertise to adapt the respective parts of the knowledge independently. We implemented this concept by integrating knowledge representation methods of the RoboEarth project with the distributed task execution capabilities of the Ubiquitous Network Robot Platform.

Today's robots are still lacking comprehensive knowledge bases about objects and their properties. Yet, a lot of knowledge is required when performing manipulation tasks to identify abstract concepts like a “handle” or the “blade of a spatula” and to ground them into concrete coordinate frames that can be used to parametrize the robot's actions. In this paper, we present a system that enables robots to use CAD models of objects as a knowledge source and to perform logical inference about object components that have automatically been identified in these models. The system includes several algorithms for mesh segmentation and geometric primitive fitting which are integrated into the robot's knowledge base as procedural attachments to the semantic representation. Bottom-up segmentation methods are complemented by top-down, knowledge-based analysis of the identified components. The evaluation on a diverse set of object models, downloaded from the Internet, shows that the algorithms are able to reliably detect several kinds of object parts.

Recently, there has been growing interest in enabling robots to use task instructions from the Internet and to share tasks they have learned with each other. To competently use, select and combine such instructions, robots need to be able to find out if different instructions describe the same task, which parts of them are similar and which ones differ. In this paper, we investigate techniques for automatically aligning symbolic task descriptions. We propose to adapt and extend established algorithms for sequence alignment that are commonly used in bioinformatics in order to make them applicable to robot action specifications. The extensions include methods for the comparison of complex sequence elements, for taking the semantic similarity of actions into account, and for aligning descriptions at different levels of granularity. We evaluate the algorithm on two large datasets of observations of human everyday tasks and show that they are able to align action sequences performed by different subjects in very different ways.

The community-based generation of content has been tremendously successful in the World-Wide Web-people help each other by providing information that could be useful to others.We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks.ROBOEARTH is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored.In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.Note to Practitioners-In this paper, we report on a formal language for knowledge representation that is used in the ROBOEARTH system, a web-based knowledge base intended to be like a "Wikipedia for robots."The objective is to enable robots to share information about how to perform actions, how to recognize and interact with objects, and where to find objects in an environment.The developed language allows to store such information in a format that supports logical inference, so that robots can for example autonomously decide if they have all prerequisites needed for performing a described action.In laboratory experiments, the system has been applied to the exchange of pick-and-place style activities between two mobile manipulation robots.We are currently extending the representation towards more fine-grained action specifications.

This paper presented an approach to create 3D object models for robotic and vision applications in a fast and inexpensive way compared to established approaches. By using the RoboEarth system for storing the created object models users have world-wide access to the data and can immediately reuse a model as soon as it was created and uploaded. The approach shows general applicability for different kinds of cameras. In this work this was shown by two example implementations for the recognition process of objects. The quality of the recognition can be verified in the video. Combined with the knowledge saved in the RoboEarth database the objects can also be properly classified.

In the context of robotic assistants in human everyday environments, pick and place tasks are beginning to be competently solved at the technical level. The question of where to place objects or where to pick them up from, among other higher-level reasoning tasks, is therefore gaining practical relevance. In this work, we consider the problem of identifying the organizational structure within an environment, i.e. the problem of determining organizational principles that would allow a robot to infer where to best place a particular, previously unseen object or where to reasonably search for a particular type of object given past observations about the allocation of objects to locations in the environment. This problem can be reasonably formulated as a classification task. We claim that organizational principles are governed by the notion of similarity and provide an empirical analysis of the importance of various features in datasets describing the organizational structure of kitchens. For the aforementioned classification tasks, we compare standard classification methods, reaching average accuracies of at least 79% in all scenarios. We thereby show that, in particular, ontology-based similarity measures are well-suited as highly discriminative features. We demonstrate the use of learned models of organizational principles in a kitchen environment on a real robot system, where the robot identifies a newly acquired item, determines a suitable location and then stores the item accordingly.

The community-based generation of content has been tremendously successful in the World Wide Web - people help each other by providing information that could be useful to others. We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks. RoboEarth is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored. In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.

In this article we investigate the representation and acquisition of Semantic Objects Maps (SOMs) that can serve as information resources for autonomous service robots performing everyday manipulation tasks in kitchen environments. These maps provide the robot with information about its operation environment that enable it to perform fetch and place tasks more efficiently and reliably. To this end, the semantic object maps can answer queries such as the following ones: “What do parts of the kitchen look like?”, “How can a container be opened and closed?”, “Where do objects of daily use belong?”, “What is inside of cupboards/drawers?”, etc. The semantic object maps presented in this article, which we call SOM+, extend the first generation of SOMs presented by Rusu et al. [1] in that the representation of SOM+is designed more thoroughly and that SOM+also include knowledge about the appearance and articulation of furniture objects. Also, the acquisition methods for SOM+substantially advance those developed in [1] in that SOM+are acquired autonomously and with low-cost (Kinect) instead of very accurate (laser-based) 3D sensors. In addition, perception methods are more general and are demonstrated to work in different kitchen environments.

Mobile manipulation robots are becoming more and more common and begin to extend their task spectrum towards more general housework activities. The sequence of actions needed to accomplish such tasks can be obtained from instructions on the Internet originally written for humans. While giving valuable information about the types of actions and some of their parameters, these instructions usually lack information that humans consider to be obvious. In this paper, we investigate how we can equip robots with sufficient knowledge and inference mechanisms to competently detect and fill such knowledge gaps in descriptions of everyday activities. We present methods for projecting the effects of actions and processes, for inferring action parameters like the objects and locations to be used, and introduce representations for reasoning about object transformations resulting from the effects of actions.

This article gives an overview of cognition-enabled robot control, a computational model for controlling autonomous service robots to achieve home chore task intelligence. For the realization of task intelligence, this computational model puts forth three core principles, which essentially involve the combination of reactive behavior specifications represented as semantically interpretable plans with inference mechanisms that enable flexible decision making. The representation of behavior specifications as plans enables the robot to not only execute the behavior specifications but also to reason about them and alter them during execution. We provide a description of a complete system for cognition-enabled robot control that implements the three core principles, demonstrating the feasibility of our approach.

We present a system for learning models of human reaching trajectories in the context of everyday manipulation activities. Different kinds of trajectories are automatically discovered, and each of them is described by its semantic context. In a first step, the system clusters trajectories in observations of human everyday activities based on their shapes, and then learns the relation between these trajectories and the contexts in which they are used. The resulting models can be used for robots to select a trajectory to use in a given context. They can also serve as powerful prediction models for human motions to improve human-robot interaction. Experiments on the TUM kitchen data set show that the method is capable of discovering meaningful clusters in real-world observations of everyday activities like setting a table.

In this work we report about our efforts to equip service robots with the capability to acquire 3D semantic maps. The robot autonomously explores indoor environments through the calculation of next best view poses, from which it assembles point clouds containing spatial and registered visual information. We apply various segmentation methods in order to generate initial hypotheses for furniture drawers and doors. The acquisition of the final semantic map makes use of the robot's proprioceptive capabilities and is carried out through the robot's interaction with the environment. We evaluated the proposed integrated approach in the real kitchen in our laboratory by measuring the quality of the generated map in terms of the map's applicability for the task at hand (e.g. resolving counter candidates by our knowledge processing system).

People perform daily activities in many different ways. When setting a table, they might use a tray, stack plates, stack cups on plates, leave the doors of a cupboard open when taking several items out of it. Similarly flexible behavior is desired when mobile robots perform household tasks. Moreover, they should perform actions in a way that they are accepted by the people, for example by showing human-like behavior. In this paper we propose to extend a transformational planning system with models characterizing the behavior produced by the different plans in the plan library. These models are used by the robot to select a plan that resembles human behavior. In addition to acting more human-like, this helps the robot choose good plans for a task by imitating humans instead of performing exhaustive search. We show the feasibility of this approach using a household robot application as an example and present empirical results on the classification accuracy in this domain.

Service robots will have to accomplish more and more complex, open-ended tasks and regularly acquire new skills. In this work, we propose a new approach to the problem of generating plans for such household robots. Instead composing them from atomic actions - the common approach in robot planning - we propose to transform task descriptions on web sites like ehow.com into executable robot plans. We present methods for automatically converting the instructions from natural language into a formal, logic-based representation, for resolving the word senses using the WordNet database and the Cyc ontology, and for exporting the generated plans into the mobile robot's plan language RPL. We discuss the problem of inferring information that is missing in these descriptions and the problem of grounding the abstract task descriptions in the perception and action system, and we propose techniques for solving them. The whole system works autonomously without human interaction. It has successfully been tested with a set of about 150 natural language directives, of which up to 80% could be correctly transformed.

This paper describes CRAM (Cognitive Robot Abstract Machine) as a software toolbox for the design, the implementation, and the deployment of cognition-enabled autonomous robots performing everyday manipulation activities. CRAM equips autonomous robots with lightweight reasoning mechanisms that can infer control decisions rather than requiring the decisions to be preprogrammed. This way CRAM-programmed autonomous robots are much more flexible, reliable, and general than control programs that lack such cognitive capabilities. CRAM does not require the whole domain to be stated explicitly in an abstract knowledge base. Rather, it grounds symbolic expressions in the knowledge representation into the perception and actuation routines and into the essential data structures of the control programs. In the accompanying video, we show complex mobile manipulation tasks performed by our household robot that were realized using the CRAM infrastructure.

This paper describes and discusses the K-COPMAN (Knowledge-enabled Cognitive Perception for Manipulation) system, which enables autonomous robots to generate symbolic representations of perceived objects and scenes and to infer answers to complex queries that require the combination of perception and knowledge processing. Using K-COPMAN, the robot can solve inference tasks such as identifying items that are likely to be missing on a breakfast table. To the programmer K-COPMAN, is presented as a logic programming system that can be queried just like a symbolic knowledge base. Internally, K-COPMAN is realized through a data structure framework together with a library of state-of-the-art perception mechanisms for mobile manipulation in human environments. Key features of K-COPMAN are that it can make a robot environment-aware and that it supports goal-directed as well as passive perceptual processing. K-COPMAN is fully integrated into an autonomous mobile manipulation robot and is realized within the open-source robot library ROS.

Knowledge processing is an essential technique for enabling autonomous robots to do the right thing to the right object in the right way. Using knowledge processing the robots can achieve more flexible and general behavior and better performance. While knowledge representation and reasoning has been a well-established research field in artificial intelligence for several decades, little work has been done to design and realize knowledge processing mechanisms for the use in the context of robotic control. In this paper, we report on KNOWROB, a knowledge processing system particularly designed for autonomous personal robots. KNOWROB is a first-order knowledge representation based on description logics that provides specific mechanisms and tools for action-centered representation, for the automated acquisition of grounded concepts through observation and experience, for reasoning about and managing uncertainty, and for fast inference - knowledge processing features that are particularly necessary for autonomous robot control.

This paper introduces the assistive kitchen as a comprehensive demonstration and challenge scenario for technical cognitive systems. We describe its hardware and software infrastructure. Within the assistive kitchen application, we select particular domain activities as research subjects and identify the cognitive capabilities needed for perceiving, interpreting, analyzing, and executing these activities as research foci. We conclude by outlining open research issues that need to be solved to realize the scenarios successfully.