As the application scenarios of mobile robots are getting more complex and challenging, scene understanding becomes increasingly crucial. A mobile robot that is supposed to operate autonomously in indoor environments must have precise knowledge about what objects are present, where they are, what their spatial extent is, and how they can be reached; i.e., information about free space is also crucial. Panoptic mapping is a powerful instrument providing such information. However, building 3D panoptic maps with high spatial resolution is challenging on mobile robots, given their limited computing capabilities. In this paper, we propose PanopticNDT – an efficient and robust panoptic mapping approach based on occupancy normal distribution transform (NDT) mapping. We evaluate our approach on the publicly available datasets Hypersim and ScanNetV2. The results reveal that our approach can represent panoptic information at a higher level of detail than other state-of-the-art approaches while enabling real-time panoptic mapping on mobile robots. Finally, we prove the real-world applicability of PanopticNDT with qualitative results in a domestic application.

Automated grasping of arbitrary objects is an essential skill for many applications such as smart manufacturing and human robot interaction. This makes grasp detection a vital skill for automated robotic systems. Recent work in model-free grasp detection uses point cloud data as input and typically outperforms the earlier work on RGB(D)-based methods. We show that RGB(D)-based methods are being underestimated due to suboptimal label encodings used for training. Using the evaluation pipeline of the GraspNet-1Billion dataset, we investigate different encodings and propose a novel encoding that significantly improves grasp detection on depth images. Additionally, we show shortcomings of the 2D rectangle grasps supplied by the GraspNet-1Billion dataset and propose a filtering scheme by which the ground truth labels can be improved significantly. Furthermore, we apply established methods for uncertainty estimation on our trained models since knowing when we can trust the model's decisions provides an advantage for real-world application. By doing so, we are the first to directly estimate uncertainties of detected grasps. We also investigate the applicability of the estimated aleatoric and epistemic uncertainties based on their theoretical properties. Additionally, we demonstrate the correlation between estimated uncertainties and grasp quality, thus improving selection of high quality grasp detections. By all these modifications, our approach using only depth images can compete with point-cloud-based approaches for grasp detection despite the lower degree of freedom for grasp poses in 2D image space.

This paper presents the results of the German research project SYMPARTNER (4/2015 - 6/2018), which aimed at developing a functional-emotional, mobile domestic robot companion for elderly people. The paper gives an overview of the developed robot, its system architecture, and essential skills and behaviors required for being a friendly home companion. Based on this, in a long-term field study running from January to June 2018 both technical aspects regarding the practical suitability and robustness of the robot under domestic operating conditions and social scientific questions on usability and acceptance of the robot and the users’ familiarization with their new housemate were evaluated. In the field study, two of these autonomous companion robots were used in 20 senior households in Erfurt (Germany). All participants lived with their robot in their apartments for one week without the need for supervising or supporting persons being present on-site. The tests in 20 single-person households in the age group 62 to 94 years (average 74 years) provided important insights into the special challenges of domesticity from a technical, social scientific, and user-oriented point of view. The results of the study show how seniors can shape their everyday life with a companion robot and how quickly they get used to the new housemate.

Due to recent deep learning techniques, person detection seems to be solved in the computer vision domain, however, it is still an issue in mobile robotics. On a robot only limited computing capacities are available. The challenge gets even more difficult when operating in an environment, with people in poses different from the standard upright ones. In this work the environment of a supermarket is considered. Unlike most scenarios targeted by the community, persons not only occur in standing postures, but also grasping into the shelves or squatting in front of them. Furthermore, people are heavily occluded, e.g. by shopping carts. In such a challenging environment, it is important to perceive people early enough and in real-time in order to enable a socially aware navigation. Classical person detectors often suffer from a high posture variance or do not achieve acceptable real-time detection rates. For this reason, different components from the 3D object detection domain have been used to create a new robust person detector for mobile application. Operating on 3D point clouds allows fast detections in real-time up to our goal distance of ten meters and above using the Kinect2 depth sensor. The detector can even differentiate between typical postures of customers who stand or squat in front of shelves.

Tracking people in the surroundings of interactive service robots is a topic of high interest. Even if image based detectors using deep learning techniques have improved the detection rate and accuracy a lot, for robotic applications it is necessary to integrate those detections over time and over the limited ranges of individual sensors into a global model. That data fusion enables a continuous state estimation of people and helps reducing the false decisions taken by individual detectors and increasing the overall range. In this paper, we present a tracking framework with a new distance measure for data association and a proper consideration of individual sensors' accuracies. By means of that, we could deal with high false detection rates of laser-based leg detectors without introducing further heuristics like a background model. The proposed system is compared to other tracking approaches from the state of the art. Furthermore, we present a novel manually annotated benchmark dataset for multi sensor person tracking from a moving robot platform in a guide scenario, which will be made publicly available.

This paper introduces a novel robot-based approach to the stroke rehabilitation scenario, in which a mobile robot companion accompanies stroke patients during their walking self-training. This assistance enables them to move freely in the clinic practicing both their mobility and spatial orientation skills. Based on a set of questions for systematic evaluating the autonomy and practicability of assistive robots and a three-stage approach in conducting function and user tests in the clinical setting, we present the results of user trials performed with N=30 stroke patients in a stroke rehabilitation center between 4/2015 and 3/2016. This allowed us to make an honest inventory of the strengths and weaknesses of the developed robot companion and its already achieved practicability for clinical use. The results of the user studies show that patients and fellow patients were very open-minded and accepted the robotic coach. The robot motivated them for independent training and leaving their room, despite severe consequences of stroke (lower limbs paralysis, speech/language problems, loss of orientation, depression), provided a very self-determined training regime, and encouraged them to expand the radius of their training in the clinic.

During social interaction between humans and robots, body language can contribute to communicate mutual emotional states. In this paper, a method is presented, that enables wheeled robots with differential drive to express various emotional states by means of different movement styles during goal-directed motion. The motion control of the robot utilizes an objective-based motion planner optimizing the control commands in a high-dimensional search space by means of an evolutionary algorithm. The main contribution in this paper is an objective function that uses human feedback on the perceived robot emotion in order to evaluate the possible robot control sequences. First group experiments have been conducted in order to demonstrate the system and find suitable movement styles for possible emotional states.

This paper presents the implementation and evaluation results of the German research project SERROGA (2012 till mid 2015), which aimed at developing a robot companion for domestic health assistance for older people that helps keeping them physically and mentally fit to remain living independently in their own homes for as long as possible. The paper gives an overview of the developed companion robot, its system architecture, and essential skills, behaviors, and services required for a robotic health assistant. Moreover, it presents a new approach allowing a quantitative description and assessment of the navigation complexity of apartments to make them objectively comparable for function tests under real-life conditions. Based on this approach, the results of function tests executed in 12 apartments of project staff and seniors are described. Furthermore, the paper presents findings of a case study conducted with nine seniors (aged 68-92) in their own homes, investigating both instrumental and social-emotional functions of a robotic health assistant. The robot accompanied the seniors in their homes for up to three days assisting with tasks of their daily schedule and health care, without any supervising person being present on-site. Results revealed that the seniors appreciated the robot's health-related instrumental functions and even built emotional bonds with it.

In this paper, a dialog modeling approach for long-term interaction between a service robot and a single user is presented, which enables a user-adaptive interaction behavior of the robot. Central element of the dialog system is a probabilistic model of the user's reactions to the robot's behavior, which is learned online and used for a probabilistic planning process based on message passing in a dynamic factor graph. The suggested approach has been applied to implement a complex application on a mobile service robot, which has been tested in a 10 day evaluation study with 16 users in order to get a feedback on usability of the interaction design, adaptation skills, and feasibility of a rapid application development. Results and findings of that study are presented here briefly.

Non-contact image photoplethysmography has gained a lot of attention during the last 5 years. Starting with the work of Verkruysse et al. [1], various methods for estimation of the human pulse rate from video sequences of the face under ambient illumination have been presented. Applied on a mobile service robot aimed to motivate elderly users for physical exercises, the pulse rate can be a valuable information in order to adapt to the users conditions. For this paper, a typical processing pipeline was implemented on a mobile robot, and a detailed comparison of methods for face segmentation was conducted, which is the key factor for robust pulse rate extraction even, if the subject is moving. A benchmark data set is introduced focusing on the amount of motion of the head during the measurement.

As report on work in progress, this paper describes the objectives and the current state of implementation of the ongoing research project ROREAS (Robotic Rehabilitation Assistant for Stroke Patients), which aims at developing a robotic rehabilitation assistant for walking and orientation exercising in self-training during clinical stroke follow-up care. This requires strongly user-centered, polite and attentive social navigation and interaction behaviors that can motivate the patients to start, continue, and regularly repeat their self-training. Against this background, the paper gives an overview of the constraints and requirements arising from the rehabilitation scenario and the operational environment, a heavily populated multi-level rehabilitation center, and presents the robot platform ROREAS which is currently used for developing the demonstrators (walking coach and orientation coach). Moreover, it gives an overview of the robot's functional system architecture and presents selected advanced navigation and HRI functionalities required for a personal robotic trainer that can successfully operate in such a challenging real-world environment, up to the results of ongoing functionality tests and upcoming user studies.

This paper presents results of user evaluations with a socially assistive robot companion for older people suffering from mild cognitive impairment (MCI) and living (alone) at home. Within the European FP7 project “CompanionAble” (2008-2012) [1], we developed assistive technologies combining a mobile robot and smart environment with the aim to support these people and assist them living in their familiar home environment. For a final evaluation, user experience studies were conducted with volunteer users who were invited to a test home where they lived and freely used the robot and integrated system over a period of two days. Services provided by the companion robot include reminders of appointments (pre-defined or added by the users themselves or their informal carer) as well as frequent recommendations to specific activities, which were listed e.g. by their family carers. Furthermore, video contact with relatives and friends, a cognitive stimulation game designed especially to counter the progress of cognitive impairments, and the possibility to store personal items with the robot are offered. Recognition of the user entering or leaving the home is triggering situation specific reminders like agenda items due during the (expected) absence, missed calls or items not to be forgotten. Continuing our previous work published in [2], this paper presents detailed description of the implemented assistive functions and results of user studies conducted during April and May 2012 in the smart house of the Dutch project partner Smart Homes in Eindhoven, The Netherlands.

This paper presents an architectural overview of a robot-based visitor information system in a university building. Two mobile robots serve as mobile information terminals providing information about the employees, labs, meeting rooms, and offices in the building and are able to guide the visitors to these points of interest. The paper focuses on the different software components needed to meet the requirements of the multi-story office building. Furthermore, the integration of a multi-hypotheses person tracker is outlined, which helps the robots to interact with the people in their near surrounding. Besides first observations on interaction, the further development is outlined as well.

This paper presents results of the development of a socially assistive home robot companion for older people suffering from mild cognitive impairment (MCI) and living (alone) at home. This work was part of the European FP7 project “CompanionAble” (2008-2012) [1] which aimed at developing assistive technologies that can support these elderly and help them to remain in their familiar living environment for as long as possible. To overcome current market entry barriers, from the start we consistently adopted a user- and application-centered development process of the companion robot and focused on three main aspects: (i) the realization of a set of mandatory functionalities to support care recipients and caregivers, (ii) a strict design and usability driven realization to increase the acceptance of the robot by the different end-user groups (the elderly, their relatives, and caregivers), and (iii) the development and component selection considering production and operational costs. In continuation of the work presented in [2], this paper describes the final implementation of the companion robot and presents latest results of functional tests and early findings of user studies recently conducted in the smart house of the Dutch project partner Smart Homes in Eindhoven, The Netherlands.

The paper is addressing several aspects of our work as part of the European FP7 project ¿CompanionAble¿ and gives an overview of the progress in developing a socially assistive home robot companion for elderly people with mild cognitive impairment (MCI) living alone at home. The spectrum of required assistive functionalities and services that have been specified by the different end-user target groups of such a robot companion (the elderly, relatives, caregivers) is manifold. It reaches from situation-specific, intelligent reminding (e.g. taking medication or drinking) and cognitive stimulation, via mobile videophony with relatives or caregivers, up to the autonomous detection of dangerous situations, like falls, and their evaluation by authorized persons via mobile telepresence. From the beginning, our approach has been focused on long-term and everyday suitability and low-cost producibility as important prerequisites for the marketability of the robot companion. Against this background, the paper presents the main system requirements derived from user studies, the consequences for the hardware design and functionality of the robot companion, its system architecture, a key technology for HRI in home environments - the autonomous user tracking and searching, up to the results of already conducted and ongoing functionality tests and upcoming user studies.

The paper gives an overview of the progress in developing a socially assistive home robot companion for elderly people with mild cognitive impairment (MCI) living alone at home. The spectrum of required assistive functionalities of such a robot companion is broad and reaches from reminding functions (e.g. taking medication or drinking) and cognitive stimulation exercises, via mobile videophony with relatives or caregivers, up to the detection and evaluation of critical situations, like falls. The paper is addressing several aspects of our work as part of the European FP7 project “CompanionAble”, as for example the developed robot hardware and its software and control architecture, the implemented skills for robust user detection and tracking and user-centered navigation in the home environment, and reports on already conducted and still ongoing functionality testings and pending usability studies with the end-user target groups (the elderly, relatives, caregivers).

This paper presents a model based system for a mobile robot to find an optimal pose for the observation of a person in indoor living environments. We define the observation pose as a combination of the camera position and view direction as well as further parameters like the aperture angle. The optimal placement of a camera is not trivial because of the high dynamic range of the scenes near windows or other bright light sources, which often results in poor image quality due to glare or hard shadows. The proposed method tries to minimize these negative effects by determining an optimal camera pose based on two major models: A spatial free space model and a representation of the lighting. In particular, a task-dependent optimization takes into account the intended purpose of the camera images, e.g. different inputs are needed for video communication with other people or for an image-processing based passive observation of the person's activities. To prove the validity of our approach, we present first experimental results comparing the chosen observation pose and resulting image with and without respect to lighting in different observation tasks.

The paper gives a comprehensive overview of our Shopping Guide project, which aims at the development of interactive mobile shopping companion robots for everyday use in challenging operating environments such as home improvement stores. It is spanning an arc from the expectations and requirements of store owners and customers, via the challenges of the shopping scenario and the operating environment, the implemented functionality of the shopping guide robots, up to the results of long-term field trials. The field trials started in April 2008 and still ongoing aim at studying whether and how a group of interactive mobile shopping guide robots can operate completely autonomously in such everyday environments and how they are accepted by uninstructed customers. In these field trials, where nine robotic shopping guides traveled together 2187 kilometers in three different home improvement stores in Germany, more than 8,600 customers were successfully guided to the locations of their products of choice. With the successful development of these shopping guide robots, a further important step towards assistive robotics for daily use has been done.

Correctly identifying people who are interested in an interaction with a mobile robot is an essential task for a smart Human-Robot Interaction. In this paper an approach is presented for selecting suitable trajectory features in a task specific manner from a huge amount of different forms of possible representations. Different sub-sampling techniques are proposed to generate trajectory sequences from which features are extracted. The trajectory data was generated in real world experiments that include extensive user interviews to acquire information about user behaviors and intentions. Using those feature vectors in a classification method enables the robot to estimate the user's interaction interest. For generating low-dimensional feature vectors, a common method, the Principle Component Analysis, is applied. The selection and combination of useful features out of a set of possible features is carried out by an information theoretic approach based on the Mutual Information and Joint Mutual Information with respect to the user's interaction interest. The introduced procedure is evaluated with neural classifiers, which are trained with the extracted features of the trajectories and the user behavior gained by observation as well as user interviewing. The results achieved indicate that an estimation of the user's interaction interest using trajectory information is feasible.

The paper describes progress achieved in our long-term research project ShopBot, which aims at the development of an intelligent and interactive mobile shopping assistant for everyday use in shopping centers or home improvement stores. It is focusing on recent progress concerning two important methodological aspects: (i) the on-line building of maps of the operation area by means of advanced Rao-Blackwellized SLAM approaches using both sonar-based gridmaps as well as vision-based graph maps as representations, and (ii) a probabilistic approach to multi-modal user detection and tracking during the guidance tour. Experimental results of both the map building characteristics and the person tracking behavior achieved in an ordinary home improvement store demonstrate the reliability of both approaches. Moreover, we present first very encouraging results of long-term field trials which have been executed with three robotic shopping assistants in another home improvement store in Bavaria since March 2008. In this field test, the robots could demonstrate their suitability for this challenging real-world application, as well as the necessary user acceptance.

In this paper, we present two important aspects of our human-robot communication interface which is being developed in the context of our long-term research framework PERSES dealing with the development of highly interactive mobile robotic assistants. First, we introduce a multi-modal people detection and tracking system, a fundamental prerequisite for the observation of a human interaction partner and his nonverbal instructions given by pointing poses, gestures, head pose and eye gaze. Based on this detection and tracking system, we present a hierarchical neural architecture that is capable of estimating a target point at the floor given a pointing pose, thus enabling a user to command his mobile robot to a specific target position in his local surroundings by means of pointing. In this context, we were especially interested in determining whether it is possible to accomplish such a target point estimator using only monocular images of low-cost cameras. Both the tracker and the target point estimator were implemented and experimentally investigated on our mobile robotic assistant HOROS. The achieved recognition results presented finally demonstrate that it is in fact possible to realize a user-independent pointing pose estimation using monocular images only, but further efforts are necessary to improve the robustness of this approach for everyday application.

For socially interactive robots it is essential to be able to estimate the interest of people to interact with them. Based on this estimation the robot can adapt its dialog strategy to the different people's behaviors. Consequently, efficient and robust techniques for people detection and tracking are basic prerequisites when dealing with human-robot interaction (HRI) in real-world scenarios. In this paper, we introduce an imposed approach for integration of several sensor modalities and present a multimodal, probability-based people detection and tracking system and its application using the different sensory systems of our mobile interaction robot HOROS. For each of these sensory cues, separate and specific Gaussian distributed hypotheses are generated and further merged into a robot-centered map by means of a flexible probabilistic aggregation scheme based on covariance intersection (CI). The main advantages of this approach are the simple extensibility by integration of further sensory channels, even with different update frequencies, and the usability in real-world HRI tasks. Finally, promising experimental results achieved for people tracking in a real-world environment, and university building, are presented

This paper describes a novel omnivision-based concurrent map-building and localization (CML) approach which is able to robustly localize a mobile robot in a uniformly structured, maze-like environment with changing appearances. The presented approach extends and improves known appearance-based CML techniques in a few essential aspects. For example, an advanced learning scheme in combination with an active forgetting is introduced to allow a complexity restricting adaptation of the environment model to appearance variations of the operation area. Moreover, a generalized scheme for fusion of localization hypotheses from several state estimators with different meaning and certainty and a distributed coding of the current observation by a weighted set of reference observations is proposed. Finally, several real-world localization experiments investigating the stability and localization accuracy of this novel omnivision-based CML technique for a highly dynamic and populated operation area, a home store, are presented.