A key proficiency an autonomous mobile robot must have to perform high-level tasks is a strong understanding of its environment. This involves information about what types of objects are present, where they are, what their spatial extend is, and how they can be reached, i.e., information about free space is also crucial. Semantic maps are a powerful instrument providing such information. However, applying semantic segmentation and building 3D maps with high spatial resolution is challenging given limited resources on mobile robots. In this paper, we incorporate semantic information into efficient occupancy normal distribution transform (NDT) maps to enable real-time semantic mapping on mobile robots. On the publicly available dataset Hypersim, we show that, due to their sub-voxel accuracy, semantic NDT maps are superior to other approaches. We compare them to the recent state-of-the-art approach based on voxels and semantic Bayesian spatial kernel inference (S-BKI) and to an optimized version of it derived in this paper. The proposed semantic NDT maps can represent semantics to the same level of detail, while mapping is 2.7 to 17.5 times faster. For the same grid resolution, they perform significantly better, while mapping is up to more than 5 times faster. Finally, we prove the real-world applicability of semantic NDT maps with qualitative results in a domestic application.

Analyzing scenes thoroughly is crucial for mobile robots acting in different environments. Semantic segmentation can enhance various subsequent tasks, such as (semantically assisted) person perception, (semantic) free space detection, (semantic) mapping, and (semantic) navigation. In this paper, we propose an efficient and robust RGB-D segmentation approach that can be optimized to a high degree using NVIDIA TensorRT and, thus, is well suited as a common initial processing step in a complex system for scene analysis on mobile robots. We show that RGB-D segmentation is superior to processing RGB images solely and that it can still be performed in real time if the network architecture is carefully designed. We evaluate our proposed Efficient Scene Analysis Network (ESANet) on the common indoor datasets NYUv2 and SUNRGB-D and show that we reach state-of-the-art performance while enabling faster inference. Furthermore, our evaluation on the outdoor dataset Cityscapes shows that our approach is suitable for other areas of application as well. Finally, instead of presenting benchmark results only, we also show qualitative results in one of our indoor application scenarios.

Efficient and robust person perception is one of the most basic skills a mobile robot must have to ensure intuitive human-machine interaction. In addition to person detection, this also includes estimating various attributes, like posture or body orientation, in order to achieve user-adaptive behavior. However, given limited computing and battery capabilities on a mobile robot, it is inefficient to solve all perception tasks separately, especially when using computationally expensive deep neural networks. Therefore, we propose a multi-task system for person perception, comprising of a fast, depth- based region proposal and an efficient, lightweight deep neural network. Using a single network forward pass, the system simultaneously detects persons, classifies their body postures, and estimates the upper body orientations while retaining almost the same computation time as a single-task network. We describe how to handle a real-world multi-task scenario and conduct an extensive series of experiments in order to compare various network architectures and task weightings. We further show that multi-task learning improves the networks' performance compared to their single-task baselines. For training and evaluation, we combine an existing dataset for orientation estimation and a new, self-recorded dataset, consisting of more than 235,000 depth patches that is made publicly available to the research community.

In this paper, we present a system for socially aware robot navigation for a wide range of service tasks in supermarkets. It comprises modules for real-time person detection and tracking to gain situation awareness, modules to react to situations, and means for human-robot communication. The technical performance of the situation awareness was evaluated in a shelf out-of-stock (SOOS) detection scenario under real-world conditions in a supermarket in Germany. Furthermore, in order to investigate whether and to what extent our social navigation strategy can improve the acceptance and application of a mobile service robot in a supermarket, we have conducted surveys with N = 60 participants and usability tests with N =8 participants during a three-day field test. We can show that a robot for SOOS detection operating in a supermarket during the opening hours is generally accepted by customers and that the integration of a real-time person perception is crucial, especially for keeping appropriate distances to persons and for improving user-centered communication. Furthermore, our results indicate that various communication channels (e.g. speech, a video projector, and LED lights) are beneficial in order to address a wider user group in the targeted supermarket setting.

In order to deploy service robots in environments where they encounter and/or cooperate with persons, one important key factor is human acceptance. Hence, information on which upcoming actions of the robot are based has to be made transparent and understandable to the human. However, considering the restricted power resources of mobile robot platforms, systems for visualization not only have to be expressive but also energy efficient. In this paper, we applied the well-known technique of laser scanning on a mobile robot to create a novel system for intention visualization and human-robot-interaction. We conducted user tests to compare our system to a low-power consuming LED video projector solution in order to evaluate the suitability for mobile platforms and to get human impressions of both systems. We can show that the presented system is preferred by most users in a dynamic test setup on a mobile platform.

Due to recent deep learning techniques, person detection seems to be solved in the computer vision domain, however, it is still an issue in mobile robotics. On a robot only limited computing capacities are available. The challenge gets even more difficult when operating in an environment, with people in poses different from the standard upright ones. In this work the environment of a supermarket is considered. Unlike most scenarios targeted by the community, persons not only occur in standing postures, but also grasping into the shelves or squatting in front of them. Furthermore, people are heavily occluded, e.g. by shopping carts. In such a challenging environment, it is important to perceive people early enough and in real-time in order to enable a socially aware navigation. Classical person detectors often suffer from a high posture variance or do not achieve acceptable real-time detection rates. For this reason, different components from the 3D object detection domain have been used to create a new robust person detector for mobile application. Operating on 3D point clouds allows fast detections in real-time up to our goal distance of ten meters and above using the Kinect2 depth sensor. The detector can even differentiate between typical postures of customers who stand or squat in front of shelves.

An essential feature for navigating socially with a mobile robot is the upper body orientation of persons in its vicinity. For example, in a supermarket orientation indicates whether a person is looking at goods on the shelves or where a person is likely to go. However, given limited computing and battery capabilities, it is not possible to rely on high-performance graphics cards to run large, computationally expensive deep neural networks for orientation estimation in real time. Nevertheless, deep learning performs quite well for regression problems. Therefore, we tackle the problem of upper body orientation estimation with small yet efficient deep neural networks on a mobile robot in this paper. We employ a fast person detection approach as preprocessing that outputs fixed size person images before the actual estimation of the orientation is done. The combination with lightweight networks allows us to estimate a continuous angle in real time, even using a CPU only. We experimentally evaluate the performance of our system on a new, self-recorded data set consisting of more than 100,000 RGB-D samples from 37 persons, which is made publicly available. We also do an extensive comparison of different network architectures and output encodings for their applicability in estimating orientations. Furthermore, we show that depth images are more suitable for the task of orientation estimation than RGB images or the combination of both.

Tracking people in the surroundings of interactive service robots is a topic of high interest. Even if image based detectors using deep learning techniques have improved the detection rate and accuracy a lot, for robotic applications it is necessary to integrate those detections over time and over the limited ranges of individual sensors into a global model. That data fusion enables a continuous state estimation of people and helps reducing the false decisions taken by individual detectors and increasing the overall range. In this paper, we present a tracking framework with a new distance measure for data association and a proper consideration of individual sensors' accuracies. By means of that, we could deal with high false detection rates of laser-based leg detectors without introducing further heuristics like a background model. The proposed system is compared to other tracking approaches from the state of the art. Furthermore, we present a novel manually annotated benchmark dataset for multi sensor person tracking from a moving robot platform in a guide scenario, which will be made publicly available.

One important function in assistive robotics for home applications is the detection of emergency cases, like falls. In this paper, we present a new detection system which can run on a mobile robot to detect persons after a fall event robustly. The system is based on 3D Normal Distributions Transform (NDT) maps on which a powerful segmentation is applied. Segments most likely belonging to a person lying on the ground are grouped into clusters. After extracting features with a soft encoding approach, each cluster is classified separately. Our experiments show that the system is able to reliably detect fallen persons in real-time. It clearly outperforms other 3D state-of-the-art approaches. We can show that our system is able to handle even very challenging situations, where fallen persons are very close to other objects in the apartment. Such complex fall events often occur in real-world applications.