The increasing usage of modern AI techniques represents a transforming shift in the robotics domain. Training and accessing new models requires substantial amounts of application-specific data, but the limited resources onboard mobile robots (like processing power, network bandwidth, etc.) pose a challenge for the development of efficient data recording and provisioning pipelines. Furthermore, accessing specific information based on a combination of spatial, temporal and semantic information is generally not supported by currently available tools. In this paper, we present a methodology which allows the efficient recording of robotic sensor data streams. We show that our approach reduces the overall time needed until the data can be served via the spatio-temporal-semantic query interface of the semantic environment representation SEEREP. We further present that the maximum sensor data rate which can be stored to disk in real-time is increased for large robotic data types like images and point clouds in comparison to frequently employed solutions within the ROS ecosystem.

Triangle mesh maps are a versatile 3D environment representation for robots to navigate in challenging indoor and outdoor environments exhibiting tunnels, hills and varying slopes. To make use of these mesh maps, methods are needed to accurately localize robots in such maps to perform essential tasks like path planning and navigation. We present Mesh ICP Localization (MICP-L), a novel and computationally efficient method for registering one or more range sensors to a triangle mesh map to continuously localize a robot in 6D, even in GPS-denied environments. We accelerate the computation of ray casting correspondences (RCC) between range sensors and mesh maps by supporting different parallel computing devices like multicore CPUs, GPUs and the latest NVIDIA RTX hardware. By additionally transforming the covariance computation into a reduction operation, we can optimize the initial guessed poses in parallel on CPUs or GPUs, making our implementation applicable in real-time on many architectures. We demonstrate the robustness of our localization approach with datasets from agricultural, aerial, and automotive domains.

Sensor simulation has emerged as a promising and powerful technique to find solutions to many real-world robotic tasks like localization and pose tracking. However, commonly used simulators have high hardware requirements and are therefore used mostly on high-end computers. In this paper, we present an approach to simulate range sensors directly on embedded hardware of mobile robots that use triangle meshes as environment map. This library, called Rmagine, allows a robot to simulate sensor data for arbitrary range sensors directly on board via ray tracing. Since robots typically only have limited computational resources, Rmagine aims at being flexible and lightweight, while scaling well even to large environment maps. It runs on several platforms like Laptops or embedded computing boards like NVIDIA Jetson by putting an unified API over the specific proprietary libraries provided by the hardware manufacturers. This work is designed to support the future development of robotic applications depending on simulation of range data that could previously not be computed in reasonable time on mobile systems.

Within the farming industry using computer vision technologies provides great support for various processes. One example is the grain harvest with a combine harvester, which requires excellent expert knowledge to reach a certain level of quality. This means the amount of byproducts including straw and hulls should be as small as possible. Inside a combine harvester several sensors are installed which deliver important information about the current harvesting state. This includes a visual based sensor that constantly delivers images to the driver about the composition of the processed grain. Normally, it is the driver’s task to decide if the quality is sufficient or if machine settings need to be adjusted to improve the outcome. Yet, resolving this task entirely manually is rather error prone, due to the high amount of fairly similar images that need to be analyzed in a short time. Therefore we designed and implemented a system that is able to automatically detect unwanted byproducts within the wheat harvest and highlight those parts inside images delivered by the visual sensor. The system itself is a combination of an automated preprocessing step and a variation of a U­Net. The preprocessing step filters unwanted byproducts and is able to automatically generate training data for the U­Net. The U­Net is lightweight, easy to train and can potentially be used onboard the combine harvester.

We present a highly efficient approach to compute continuous shortest path vector fields on arbitrarily shaped 3D triangular meshes for robot navigation in complex real-world outdoor environments. The continuity of the vector field allows to query the shortest distance, direction and geodesic path to the goal at any point within the mesh triangles, resulting in accurate paths. In order to avoid impassable areas, our wavefront propagation method runs on a modular extendable multilayer map architecture taking different geometric cost layers into account. We describe the mathematical foundation of the geodesic distances and continuous vector field computation and demonstrate the performance in real-world and multilevel environments on our campus with a tunnel, ramps and stair- cases, and in a difficult, steep forest area with a stone quarry. For reproducibility, we provide a ready-to-use ROS software stack as well as Gazebo simulations.

While a great variety of 3D cameras have been introduced in recent years, most publicly available datasets for object recognition and pose estimation focus on one single camera. In this work, we present a dataset of 32 scenes that have been captured by 7 different 3D cameras, totaling 49,294 frames. This allows evaluating the sensitivity of pose estimation algorithms to the specifics of the used camera and the development of more robust algorithms that are more independent of the camera model. Vice versa, our dataset enables researchers to perform a quantitative comparison of the data from several different cameras and depth sensing technologies and evaluate their algorithms before selecting a camera for their specific task. The scenes in our dataset contain 20 different objects from the common benchmark YCB object and model set [1], [2]. We provide full ground truth 6DoF poses for each object, per-pixel segmentation, 2D and 3D bounding boxes and a measure of the amount of occlusion of each object. We have also performed an initial evaluation of the cameras using our dataset on a state-of-the-art object recognition and pose estimation system [3].

We present Move Base Flex (MBF), a highly flexible, modular, map-independent, open-source navigation framework for use in ROS. MBF provides modular actions for executing plugins for path planning, motion control, and recovery. These actions define interfaces for external executives to allow highly flexible navigation strategies, which can be intertwined with other robot tasks. MBF has been successfully deployed in a professional setting at customer facilities to control robots in highly dynamic environments. We compare MBF with the well-known move_base and present the architecture as well as different deployment approaches, including how MBF can be used with different executives to perform complex navigation tasks interleaved with other robot operations.

Real-world robotic systems have to perform reliably in uncertain and dynamic environments. State-of-the-art cognitive robotic systems use an abstract symbolic representation of the real world for high-level reasoning. Some aspects of the world, such as object dynamics, are inherently difficult to capture in an abstract symbolic form, yet they influence whether the executed action will succeed or fail. This paper presents an integrated system that uses a physics-based simulation to predict robot action results and durations, combined with a Hierarchical Task Network (HTN) planner and semantic execution monitoring. We describe a fully integrated system in which a Semantic Execution Monitor (SEM) uses information from the planning domain to perform functional imagination. Based on information obtained from functional imagination, the robot control system decides whether it is necessary to adapt the plan currently being executed. As a proof of concept, we demonstrate a PR2 able to carry tall objects on a tray without the objects toppling. Our approach achieves this by simulating robot and object dynamics. A validation shows that robot action results in simulation can be transferred to the real world. The system improves on state-of-the-art AI plan-based systems by feeding simulated prediction results back into the execution system.

Plan-based robot control has to consider a multitude of aspects of tasks at once, such as task dependency, time, space, and resource usage. Hybrid planning is a strategy for treating them jointly. However, by incorporating all these aspects into a hybrid planner, its search space is huge by construction. This paper introduces the planner CHIMP, which is based on meta-CSP planning to represent the hybrid plan space and uses hierarchical planning as the strategy for cutting efficiently through this space. The paper makes two contributions: First, it describes how HTN planning is integrated into meta-CSP reasoning leading to a planner that can reason about different forms of knowledge and that is fast enough to be used on a robot. Second, it demonstrates CHIMP's task merging capabilities, i.e., the unification of different tasks from different plan parts, resulting in plans that are more efficient to execute. It also allows to merge new tasks online into a plan that is being executed. This is demonstrated on a PR2 robot.

We present an approach to create a semantic map of an indoor environment, based on a series of 3D point clouds captured by a mobile robot using a Kinect camera. The proposed system reconstructs the surfaces in the point clouds, detects different types of furniture and estimates their poses. The result is a consistent mesh representation of the environment enriched by CAD models corresponding to the detected pieces of furniture. We evaluate our approach on two datasets totaling over 800 frames directly on each individual frame.

Textured polygon meshes are becoming more and more important for robotic applications. In this paper we present an approach to automatically extract textures from colored 3D point cloud data and apply them to a polygonal reconstruction of the scene. The extracted textures are analyzed for existing patters and reused if several instances appear. Emphasis of this work is on minimizing the number of used pixels while maintaining a realistic impression of the scanned environment.

To advance robotic science it is important to perform experiments that can be replicated by other researchers to compare different methods. However, these comparisons tend to be biased, since re-implementations of reference methods often lack thoroughness and do not include the hands-on experience obtained during the original development process. This paper presents a thorough comparison of 3D scan registration algorithms based on a 3D mapping field experiment, carried out by two research groups that are leading in the field of 3D robotic mapping. The iterative closest points algorithm (ICP) is compared to the normal distributions transform (NDT). We also present an improved version of NDT with a substantially larger valley of convergence than previously published versions.

In the past many solutions for simultaneous localization and mapping (SLAM) have been presented. Recently these solutions have been extended to map large environments with six degrees of freedom (DoF) poses. To demonstrate the capabilities of these SLAM algorithms it is common practice to present the generated maps and successful loop closing. Unfortunately there is often no objective performance metric that allows to compare different approaches. This fact is attributed to the lack of ground truth data. For this reason we present a novel method that is able to generate this ground truth data based on reference maps. Further on, the resulting reference path is used to measure the absolute performance of different 6D SLAM algorithms building a large urban outdoor map.

To create with an autonomous mobile robot a 3D volumetric map of a scene it is necessary to gage several 3D scans and to merge them into one consistent 3D model. This paper provides a new solution to the simultaneous localization and mapping (SLAM) problem with six degrees of freedom. Robot motion on natural surfaces has to cope with yaw, pitch and roll angles, turning pose estimation into a problem in six mathematical dimensions. A fast variant of the Iterative Closest Points algorithm registers the 3D scans in a common coordinate system and relocalizes the robot. Finally, consistent 3D maps are generated using a global relaxation. The algorithms have been tested with 3D scans taken in the Mathies mine, Pittsburgh, PA. Abandoned mines pose significant problems to society, yet a large fraction of them lack accurate 3D maps.

This paper presents a robust and real-time capable recognition system for the fast detection and classification of objects in spatial 3D data. Depth and reflection data from a 3D laser scanner are rendered into images and fed into a saliency-based visual attention system that detects regions of potential interest. Only these regions are examined by a fast classifier. The time saving of classifying objects in salient regions rather than in complete images is linear with the number of trained object classes. Robustness is achieved by the fusion of the bi-modal scanner data; in contrast to camera images, this data is completely illumination independent. The recognition system is trained for two different object classes and evaluated on real indoor data.

This paper describes a novel, laser-based approach for tracking the pose of a high-speed mobile robot. The algorithm is outstanding in terms of accuracy and computational time, being 33 times faster than real time. The efficiency is achieved by a closed form solution for the matching of two lasers scans, the use of natural landmarks and fast linear filters. The implemented algorithm is evaluated with the high-speed robot Kurt3D (4 m/s), and compared to standard scan matching methods in indoor and outdoor environments.

High-level symbolic representations of actions to control the working of autonomous robots are used in all hybrid (reactive and deliberative) robot control architectures. Abstract action representations serve several purposes, such as structuring the control code, optimizing the robot performance, and providing a basis for reasoning about future robot action. The paper presents results about re-designing the RHINO navigation system by introducing an HTN plan layer. Besides yielding a more structured robot control software, this layer is used as a basis for optimizing the navigation performance by plan transformations. We show how a robot can learn to select plan transformations based on projections of its intended behavior. Our experimental evaluation shows that the overall robot navigation performance is increased by almost 42 % when using learned projective models to select plan transformations.

This work presents the concept to recover and utilize the visual context in panoramic images. Omnidirectional imaging has become recently an efficient basis for robot navigation. The proposed Bayesian reasoning over local image appearances enables to reject false hypotheses which do not fit the structural constraints in corresponding feature trajectories. The methodology is proved with real image data from an office robot to dramatically increase the localization performance in the presence of severe occlusion effects, particularly in noisy environments, and to recover rotational information on the fly.

We present a technique for extending a given metric map of the environment by objects of a known type, where localization and perception of the robot is allowed to be uncertain. The advantage of our approach is that it allows the robot to estimate its own position in the given outline of the environment and thus to estimate the position of the objects not contained in the map. The method relies on partially observable Markov decision processes as well as on the Baum-Welch algorithm. It has been implemented and evaluated in several simulation experiments and also in a real-world sewage pipe network. The experimental results demonstrate that our approach can efficiently and accurately estimate the position of unexpected objects. Due to the probabilistic nature of the underlying techniques, our method can deal with noisy sensors as well as with large odometry errors which generally occur when deploying a robot in a sewerage pipe system.

Building planning systems that operate in real domains requires coping with both uncertainty and time pressure. This article describes a model of reaction plans, which are generated using a formalization of actions and of state descriptions in probabilistic logic, as a basis for anytime planning under uncertainty. the model has the following main features. At the action level, we handle incomplete and ambiguous domain information, and reason about alternative action effects whose probabilities are given. On this basis, we generate reaction plans that specify different courses of action, reflecting the domain uncertainty and alternative action effects; if generation time was insufficient, these plans may be left unfinished, but they can be reused, incrementally improved, and finished later. At the planning level, we develop a framework for measuring the quality of plans that takes domain uncertainty and probabilistic information into account using Markov chain theory; based on this framework, one can design anytime algorithms focusing on those parts of an unfinished plan first, whose completion promises the most “gain”. Finally, the plan quality can be updated during execution, according to additional information acquired, and can therefore be used for on-line planning. © 1995 John Wiley & Sons, Inc.

The paper describes a case study that explores the idea of building a planner with a neat semantics of the plans it produces, by choosing some action formalism that is ‘ideal’ for the planning application and building the planner accordingly. In general—and particularly so for the action formalism used in this study, which is quite expressive—this strategy is unlikely to yield fast and efficient planners if the formalism is used naïvely. Therefore, we adopt the idea that the planner approximates the theoretically ideal plans, where the approximation gets closer the more run time the planner is allowed. As the particular formalism underlying our study allows a significant degree of uncertainty to be modelled and copes with the ramification problem, we end up in a planner that is functionally comparable to modem anytime uncertainty planners, yet is based on a neat formal semantics.

The paper begins with a critique of the Ginsberg/Smith approach to reasoning about action which is based on the notion of maximal consistent subsets of world descriptions. There are some deep difficulties with this approach due to a misconception of what is possible and what is not. Incomplete information about the actual state of the world may lead to counterintuitive conclusions about what is true after an action is performed. These problems were first discussed by Winslett, who presents an alternative formalization based on possible models. Her formalization solves problems with incomplete descriptions of the world. However, related problems caused by using disjunctive postconditions to model ambiguous actions remain unsolved. Moreover, Winslett does not take the effects of causality into account when determining the closeness of worlds. We present a novel formalization of actions that avoids these problems.