Older adults, a growing demographic, face an increased risk of experiencing loneliness and are less exposed to emerging communication technologies. Augmented reality (AR) avatars and telepresence robots have been proposed as tools to foster social connection, yet their suitability for older users remains underexplored. We present an exploratory study with ten healthy older adults who engaged in both conversational and spatial collaboration tasks using AR avatar-mediated communication, robot-mediated communication, and face-to-face interaction. We collected self-reported measures of co-presence, social presence, closeness, uncanny valley, preferences, and open feedback. Our findings suggest that telepresence robots enhanced co-presence, while avatars were valued for their expressivity and humanlike qualities. Task type influenced co-presence in spatial collaboration only during communication using the telepresence robot. Other measures, such as social presence and closeness, were unaffected by task type or representation. While neither technology outperformed face-to-face interaction, both were positively received, underscoring their potential to address the social needs of older adults and highlighting the importance of enhancing nonverbal expressivity, particularly nonverbal cues in mediated communication. Ultimately, our results contribute to the fundamental understanding of mediated communication with older adults, motivating further empirical work to confirm and extend these findings.

Telepresence robots that allow communication between older adults and their remotely located social contacts can foster social integration. The present laboratory test study explores older adults’ successful use of a telepresence robot (Research Question 1 [RQ1]), as well as their perceived enjoyment (RQ2), perceived ease of use (RQ3), perceived usefulness (RQ4), perceived social presence (RQ5), and intention to use (RQ6) a telepresence robot for robot-mediated communication (RMC). Semi-structured interviews, observations, and questionnaires were applied with a group of N = 14 older adults living in Germany. Participants completed a navigational task (as remote users) and an interpersonal communication task (as local users). Results show older adults used the telepresence robot successfully (RQ1) during the tasks. Furthermore, in interviews, older adults described their perceived enjoyment (RQ2), perceived ease of use (RQ3), and perceived usefulness (RQ4) during RMC as generally high. Perceived social presence (RQ5) during RMC was generally described as high, with RMC being considered a viable substitute when face-to-face communication is not possible. Finally, only two participants (2/14) had no intention to use (RQ6) a telepresence robot in the long term. Future design recommendations are provided, such as adapting the telepresence robot’s functions to older adults physical, psychological, and social conditions.

Effective human action recognition is widely used for cobots in Industry 4.0 to assist in assembly tasks. However, conventional skeleton-based methods often lose keypoint semantics, limiting their effectiveness in complex interactions. In this work, we introduce a novel approach to skeleton-based action recognition that enriches input representations by leveraging word embeddings to encode semantic information. Our method replaces one-hot encodings with semantic volumes, enabling the model to capture meaningful relationships between joints and objects. Through extensive experiments on multiple assembly datasets, we demonstrate that our approach significantly improves classification performance, and enhances generalization capabilities by simultaneously supporting different skeleton types and object classes. Our findings highlight the potential of incorporating semantic information to enhance skeleton-based action recognition in dynamic and diverse environments.

In domestic environments, robots require a comprehensive understanding of their surroundings to interact effectively and intuitively with untrained humans. In this paper, we propose DVEFormer – an efficient RGB-D Transformer-based approach that predicts dense text-aligned visual embeddings (DVE) via knowledge distillation. Instead of directly performing classical semantic segmentation with fixed predefined classes, our method uses teacher embeddings from Alpha-CLIP to guide our efficient student model DVEFormer in learning fine-grained pixel-wise embeddings. While this approach still enables classical semantic segmentation, e.g., via linear probing, it further enables flexible text-based querying and other applications, such as creating comprehensive 3D maps. Evaluations on common indoor datasets demonstrate that our approach achieves competitive performance while meeting real-time requirements, operating at 26.3FPS for the full model and 77.0FPS for a smaller variant on an NVIDIA Jetson AGX Orin. Additionally, we show qualitative results that highlight the effectiveness and possible use cases in real-world applications. Overall, our method serves as a drop-in replacement for traditional segmentation approaches while enabling flexible natural-language querying and seamless integration into 3D mapping pipelines for mobile robotics.

Avatars and telepresence robots have long received attention for remote communication. However, the specific nature of their physicality, expressiveness, and mobility may affect their usefulness for different tasks. This work compares using an avatar (presented in augmented reality) and a telepresence robot to Face-to-Face (F2F) communication during different communication tasks: free conversation, negotiation, and referential communication with movement. We conducted a user study (split-plot design, N=54) with the type of representation of the conversational partner as the within variable and the communication task as the between variable. Our results show that the type of task, especially referential communication with movement, influenced the perceived attention to nonverbal cues and closeness. Generally, gestures and body movements received the least focus with telepresence robots. Gestures in avatars and F2F drew similar attention, which we attribute to the avatar's tracking fidelity. Gaze received less attention in both avatar- and robot-mediated communication compared to F2F, while facial expressions on the robot's screen heightened attention compared to avatars. These findings advance the fundamental understanding of mediated communication and support researchers and practitioners in shaping the design of communication applications beyond today's video calls.

Telepresence robots can enhance communication experiences by providing a sense of physical presence, embodiment and may evoke co-presence. In spite of that, telepresence robots have not made it fully to consumer markets. In this paper, we investigate how different levels of controlling a telepresence robot (teleoperation, shared control, and no control) influence presence. To this aim, we conducted a study (N=45) where participants were evenly distributed to one of the robot control conditions. The task involved navigating an unknown room and listening to stories told by a person co-located with the robot. We collected subjective impressions of presence using the temple presence inventory and performed a thematic content analysis on a post-experiment interview. Our results suggest nuances in perceived presence under different levels of robot control after performing a thematic content analysis. Copresence can be experienced during teleoperation and shared control, and teleoperation may evoke negative sentiments if it does not provide enough spatial information during navigation. However, our results did not point to significant differences in spatial or social presence. We consider that these findings encourage further discussions on how presence is perceived in robot-mediated communication.

Humans are able to learn to recognize new objects even from a few examples. In contrast, training deep-learning-based object detectors requires huge amounts of annotated data. To avoid the need to acquire and annotate these huge amounts of data, few-shot object detection (FSOD) aims to learn from few object instances of new categories in the target domain. In this survey, we provide an overview of the state of the art in FSOD. We categorize approaches according to their training scheme and architectural layout. For each type of approach, we describe the general realization as well as concepts to improve the performance on novel categories. Whenever appropriate, we give short takeaways regarding these concepts in order to highlight the best ideas. Eventually, we introduce commonly used datasets and their evaluation protocols and analyze the reported benchmark results. As a result, we emphasize common challenges in evaluation and identify the most promising current trends in this emerging field of FSOD.

With the emergence of collaborative robots (cobots), human-robot collaboration in industrial manufacturing is coming into focus. For a cobot to act autonomously and as an assistant, it must understand human actions during assembly. To effectively train models for this task, a dataset containing suitable assembly actions in a realistic setting is cru-cial. For this purpose, we present the ATTACH dataset, which contains 51.6 hours of assembly with 95.2k annotated fine-grained actions monitored by three cameras, which represent potential viewpoints of a cobot. Since in an assembly context workers tend to perform different actions simultaneously with their two hands, we annotated the performed actions for each hand separately. Therefore, in the ATTACH dataset, more than 68% of annotations overlap with other annotations, which is many times more than in related datasets, typically featuring more simplistic assembly tasks. For better generalization with respect to the background of the working area, we did not only record color and depth images, but also used the Azure Kinect body tracking SDK for estimating 3D skeletons of the worker. To create a first baseline, we report the performance of state-of-the-art methods for action recognition as well as action detection on video and skeleton-sequence inputs. The dataset is available at https://www.tu-ilmenau.de/neurob/data-sets-code/attach-dataset.

Person re-identification plays a key role in applications where a mobile robot needs to track its users over a long period of time, even if they are partially unobserved for some time, in order to follow them or be available on demand. In this context, deep-learning-based real-time feature extraction on a mobile robot is often performed on special-purpose devices whose computational resources are shared for multiple tasks. Therefore, the inference speed has to be taken into account. In contrast, person re-identification is often improved by architectural changes that come at the cost of significantly slowing down inference. Attention blocks are one such example. We will show that some well-performing attention blocks used in the state of the art are subject to inference costs that are far too high to justify their use for mobile robotic applications. As a consequence, we propose an attention block that only slightly affects the inference speed while keeping up with much deeper networks or more complex attention blocks in terms of re-identification accuracy. We perform extensive neural architecture search to derive rules at which locations this attention block should be integrated into the architecture in order to achieve the best trade-off between speed and accuracy. Finally, we confirm that the best performing configuration on a re-identification benchmark also performs well on an indoor robotic dataset.

As the use of collaborative robots (cobots) in industrial manufacturing continues to grow, human action recognition for effective human-robot collaboration becomes increasingly important. This ability is crucial for cobots to act autonomously and assist in assembly tasks. Recently, skeleton-based approaches are often used as they tend to generalize better to different people and environments. However, when processing skeletons alone, information about the objects a human interacts with is lost. Therefore, we present a novel approach of integrating object information into skeleton-based action recognition. We enhance two state-of-the-art methods by treating object centers as further skeleton joints. Our experiments on the assembly dataset IKEA ASM show that our approach improves the performance of these state-of-the-art methods to a large extent when combining skeleton joints with objects predicted by a state-of-the-art instance segmentation model. Our research sheds light on the benefits of combining skeleton joints with object information for human action recognition in assembly tasks. We analyze the effect of the object detector on the combination for action classification and discuss the important factors that must be taken into account.

Scene analysis is essential for enabling autonomous systems, such as mobile robots, to operate in real-world environments. However, obtaining a comprehensive understanding of the scene requires solving multiple tasks, such as panoptic segmentation, instance orientation estimation, and scene classification. Solving these tasks given limited computing and battery capabilities on mobile platforms is challenging. To address this challenge, we introduce an efficient multi-task scene analysis approach, called EMSAFormer, that uses an RGB-D Transformer-based encoder to simultaneously perform the aforementioned tasks. Our approach builds upon the previously published EMSANet. However, we show that the dual CNN-based encoder of EMSANet can be replaced with a single Transformer-based encoder. To achieve this, we investigate how information from both RGB and depth data can be effectively incorporated in a single encoder. To accelerate inference on robotic hardware, we provide a custom NVIDIA TensorRT extension enabling highly optimization for our EMSAFormer approach. Through extensive experiments on the commonly used indoor datasets NYUv2, SUNRGB-D, and ScanNet, we show that our approach achieves state-of-the-art performance while still enabling inference with up to 39.1 FPS on an NVIDIA Jetson AGX Orin 32 GB.

As the application scenarios of mobile robots are getting more complex and challenging, scene understanding becomes increasingly crucial. A mobile robot that is supposed to operate autonomously in indoor environments must have precise knowledge about what objects are present, where they are, what their spatial extent is, and how they can be reached; i.e., information about free space is also crucial. Panoptic mapping is a powerful instrument providing such information. However, building 3D panoptic maps with high spatial resolution is challenging on mobile robots, given their limited computing capabilities. In this paper, we propose PanopticNDT – an efficient and robust panoptic mapping approach based on occupancy normal distribution transform (NDT) mapping. We evaluate our approach on the publicly available datasets Hypersim and ScanNetV2. The results reveal that our approach can represent panoptic information at a higher level of detail than other state-of-the-art approaches while enabling real-time panoptic mapping on mobile robots. Finally, we prove the real-world applicability of PanopticNDT with qualitative results in a domestic application.

A key proficiency an autonomous mobile robot must have to perform high-level tasks is a strong understanding of its environment. This involves information about what types of objects are present, where they are, what their spatial extend is, and how they can be reached, i.e., information about free space is also crucial. Semantic maps are a powerful instrument providing such information. However, applying semantic segmentation and building 3D maps with high spatial resolution is challenging given limited resources on mobile robots. In this paper, we incorporate semantic information into efficient occupancy normal distribution transform (NDT) maps to enable real-time semantic mapping on mobile robots. On the publicly available dataset Hypersim, we show that, due to their sub-voxel accuracy, semantic NDT maps are superior to other approaches. We compare them to the recent state-of-the-art approach based on voxels and semantic Bayesian spatial kernel inference (S-BKI) and to an optimized version of it derived in this paper. The proposed semantic NDT maps can represent semantics to the same level of detail, while mapping is 2.7 to 17.5 times faster. For the same grid resolution, they perform significantly better, while mapping is up to more than 5 times faster. Finally, we prove the real-world applicability of semantic NDT maps with qualitative results in a domestic application.

Semantic scene understanding is essential for mobile agents acting in various environments. Although semantic segmentation already provides a lot of information, details about individual objects as well as the general scene are missing but required for many real-world applications. However, solving multiple tasks separately is expensive and cannot be accomplished in real time given limited computing and battery capabilities on a mobile platform. In this paper, we propose an efficient multi-task approach for RGB-D scene analysis (EMSANet) that simultaneously performs semantic and instance segmentation (panoptic segmentation), instance orientation estimation, and scene classification. We show that all tasks can be accomplished using a single neural network in real time on a mobile platform without diminishing performance - by contrast, the individual tasks are able to benefit from each other. In order to evaluate our multi-task approach, we extend the annotations of the common RGB-D indoor datasets NYUv2 and SUNRGB-D for instance segmentation and orientation estimation. To the best of our knowledge, we are the first to provide results in such a comprehensive multi-task setting for indoor scene analysis on NYUv2 and SUNRGB-D.

Automated grasping of arbitrary objects is an essential skill for many applications such as smart manufacturing and human robot interaction. This makes grasp detection a vital skill for automated robotic systems. Recent work in model-free grasp detection uses point cloud data as input and typically outperforms the earlier work on RGB(D)-based methods. We show that RGB(D)-based methods are being underestimated due to suboptimal label encodings used for training. Using the evaluation pipeline of the GraspNet-1Billion dataset, we investigate different encodings and propose a novel encoding that significantly improves grasp detection on depth images. Additionally, we show shortcomings of the 2D rectangle grasps supplied by the GraspNet-1Billion dataset and propose a filtering scheme by which the ground truth labels can be improved significantly. Furthermore, we apply established methods for uncertainty estimation on our trained models since knowing when we can trust the model's decisions provides an advantage for real-world application. By doing so, we are the first to directly estimate uncertainties of detected grasps. We also investigate the applicability of the estimated aleatoric and epistemic uncertainties based on their theoretical properties. Additionally, we demonstrate the correlation between estimated uncertainties and grasp quality, thus improving selection of high quality grasp detections. By all these modifications, our approach using only depth images can compete with point-cloud-based approaches for grasp detection despite the lower degree of freedom for grasp poses in 2D image space.

Robust point cloud registration in real-time is an important prerequisite for many mapping and localization algorithms. Traditional methods like ICP tend to fail without good initialization, insufficient overlap or in the presence of dynamic objects. Modern deep learning based registration approaches present much better results, but suffer from a heavy runtime. We overcome these drawbacks by introducing StickyPillars, a fast, accurate and extremely robust deep middle-end 3D feature matching method on point clouds. It uses graph neural networks and performs context aggregation on sparse 3D key-points with the aid of transformer based multi-head self and cross-attention. The network output is used as the cost for an optimal transport problem whose solution yields the final matching probabilities. The system does not rely on hand crafted feature descriptors or heuristic matching strategies. We present state-of-art art accuracy results on the registration problem demonstrated on the KITTI dataset while being four times faster then leading deep methods. Furthermore, we integrate our matching system into a LiDAR odometry pipeline yielding most accurate results on the KITTI odometry dataset. Finally, we demonstrate robustness on KITTI odometry. Our method remains stable in accuracy where state-of-the-art procedures fail on frame drops and higher speeds.

Analyzing scenes thoroughly is crucial for mobile robots acting in different environments. Semantic segmentation can enhance various subsequent tasks, such as (semantically assisted) person perception, (semantic) free space detection, (semantic) mapping, and (semantic) navigation. In this paper, we propose an efficient and robust RGB-D segmentation approach that can be optimized to a high degree using NVIDIA TensorRT and, thus, is well suited as a common initial processing step in a complex system for scene analysis on mobile robots. We show that RGB-D segmentation is superior to processing RGB images solely and that it can still be performed in real time if the network architecture is carefully designed. We evaluate our proposed Efficient Scene Analysis Network (ESANet) on the common indoor datasets NYUv2 and SUNRGB-D and show that we reach state-of-the-art performance while enabling faster inference. Furthermore, our evaluation on the outdoor dataset Cityscapes shows that our approach is suitable for other areas of application as well. Finally, instead of presenting benchmark results only, we also show qualitative results in one of our indoor application scenarios.

Efficient and robust person perception is one of the most basic skills a mobile robot must have to ensure intuitive human-machine interaction. In addition to person detection, this also includes estimating various attributes, like posture or body orientation, in order to achieve user-adaptive behavior. However, given limited computing and battery capabilities on a mobile robot, it is inefficient to solve all perception tasks separately, especially when using computationally expensive deep neural networks. Therefore, we propose a multi-task system for person perception, comprising of a fast, depth- based region proposal and an efficient, lightweight deep neural network. Using a single network forward pass, the system simultaneously detects persons, classifies their body postures, and estimates the upper body orientations while retaining almost the same computation time as a single-task network. We describe how to handle a real-world multi-task scenario and conduct an extensive series of experiments in order to compare various network architectures and task weightings. We further show that multi-task learning improves the networks' performance compared to their single-task baselines. For training and evaluation, we combine an existing dataset for orientation estimation and a new, self-recorded dataset, consisting of more than 235,000 depth patches that is made publicly available to the research community.

In this paper, we present a system for socially aware robot navigation for a wide range of service tasks in supermarkets. It comprises modules for real-time person detection and tracking to gain situation awareness, modules to react to situations, and means for human-robot communication. The technical performance of the situation awareness was evaluated in a shelf out-of-stock (SOOS) detection scenario under real-world conditions in a supermarket in Germany. Furthermore, in order to investigate whether and to what extent our social navigation strategy can improve the acceptance and application of a mobile service robot in a supermarket, we have conducted surveys with N = 60 participants and usability tests with N =8 participants during a three-day field test. We can show that a robot for SOOS detection operating in a supermarket during the opening hours is generally accepted by customers and that the integration of a real-time person perception is crucial, especially for keeping appropriate distances to persons and for improving user-centered communication. Furthermore, our results indicate that various communication channels (e.g. speech, a video projector, and LED lights) are beneficial in order to address a wider user group in the targeted supermarket setting.

Successful rehabilitation after surgery in hip endoprosthetics comprises self-training of the lessons taught by physiotherapists. While doing so, immediate feedback to the patient about deviations from physiological gait patterns during training is very beneficial. In the research project ROGER, a mobile socially assistive robot (SAR), which supports patients after surgery in hip endoprosthetics during their self-training, was developed. The robot employs task-specific, user-centered navigation and autonomous, real-time gait feature classification techniques to enrich the self-training through companionship and timely corrective feedback. This paper presents technical and usability results obtained during four weeks of user tests at our partner hospital "Waldkliniken Eisenberg" in Germany.

In order to deploy service robots in environments where they encounter and/or cooperate with persons, one important key factor is human acceptance. Hence, information on which upcoming actions of the robot are based has to be made transparent and understandable to the human. However, considering the restricted power resources of mobile robot platforms, systems for visualization not only have to be expressive but also energy efficient. In this paper, we applied the well-known technique of laser scanning on a mobile robot to create a novel system for intention visualization and human-robot-interaction. We conducted user tests to compare our system to a low-power consuming LED video projector solution in order to evaluate the suitability for mobile platforms and to get human impressions of both systems. We can show that the presented system is preferred by most users in a dynamic test setup on a mobile platform.

This paper presents the results of the German research project SYMPARTNER (4/2015 - 6/2018), which aimed at developing a functional-emotional, mobile domestic robot companion for elderly people. The paper gives an overview of the developed robot, its system architecture, and essential skills and behaviors required for being a friendly home companion. Based on this, in a long-term field study running from January to June 2018 both technical aspects regarding the practical suitability and robustness of the robot under domestic operating conditions and social scientific questions on usability and acceptance of the robot and the users’ familiarization with their new housemate were evaluated. In the field study, two of these autonomous companion robots were used in 20 senior households in Erfurt (Germany). All participants lived with their robot in their apartments for one week without the need for supervising or supporting persons being present on-site. The tests in 20 single-person households in the age group 62 to 94 years (average 74 years) provided important insights into the special challenges of domesticity from a technical, social scientific, and user-oriented point of view. The results of the study show how seniors can shape their everyday life with a companion robot and how quickly they get used to the new housemate.

Due to recent deep learning techniques, person detection seems to be solved in the computer vision domain, however, it is still an issue in mobile robotics. On a robot only limited computing capacities are available. The challenge gets even more difficult when operating in an environment, with people in poses different from the standard upright ones. In this work the environment of a supermarket is considered. Unlike most scenarios targeted by the community, persons not only occur in standing postures, but also grasping into the shelves or squatting in front of them. Furthermore, people are heavily occluded, e.g. by shopping carts. In such a challenging environment, it is important to perceive people early enough and in real-time in order to enable a socially aware navigation. Classical person detectors often suffer from a high posture variance or do not achieve acceptable real-time detection rates. For this reason, different components from the 3D object detection domain have been used to create a new robust person detector for mobile application. Operating on 3D point clouds allows fast detections in real-time up to our goal distance of ten meters and above using the Kinect2 depth sensor. The detector can even differentiate between typical postures of customers who stand or squat in front of shelves.

Aging public roads need frequent inspections in order to guarantee their permanent availability. In many countries, this includes the standardized visual assessment of millions of images. Due to the lack of sophisticated approaches, often, the evaluation is done manually and therefore requires excessive manual labor. GAPs is the most extensive publicly available dataset that provides standardized, high-quality images for training deep neural networks for pavement distress detection. We further enlarge this dataset and provide refined annotations. By conducting extensive experiments on the GAPs dataset, we improve the performance of automated visual road condition assessment. We evaluate the performance gain of several modern neural network architectures and advanced training techniques.

An essential feature for navigating socially with a mobile robot is the upper body orientation of persons in its vicinity. For example, in a supermarket orientation indicates whether a person is looking at goods on the shelves or where a person is likely to go. However, given limited computing and battery capabilities, it is not possible to rely on high-performance graphics cards to run large, computationally expensive deep neural networks for orientation estimation in real time. Nevertheless, deep learning performs quite well for regression problems. Therefore, we tackle the problem of upper body orientation estimation with small yet efficient deep neural networks on a mobile robot in this paper. We employ a fast person detection approach as preprocessing that outputs fixed size person images before the actual estimation of the orientation is done. The combination with lightweight networks allows us to estimate a continuous angle in real time, even using a CPU only. We experimentally evaluate the performance of our system on a new, self-recorded data set consisting of more than 100,000 RGB-D samples from 37 persons, which is made publicly available. We also do an extensive comparison of different network architectures and output encodings for their applicability in estimating orientations. Furthermore, we show that depth images are more suitable for the task of orientation estimation than RGB images or the combination of both.

Tracking people in the surroundings of interactive service robots is a topic of high interest. Even if image based detectors using deep learning techniques have improved the detection rate and accuracy a lot, for robotic applications it is necessary to integrate those detections over time and over the limited ranges of individual sensors into a global model. That data fusion enables a continuous state estimation of people and helps reducing the false decisions taken by individual detectors and increasing the overall range. In this paper, we present a tracking framework with a new distance measure for data association and a proper consideration of individual sensors' accuracies. By means of that, we could deal with high false detection rates of laser-based leg detectors without introducing further heuristics like a background model. The proposed system is compared to other tracking approaches from the state of the art. Furthermore, we present a novel manually annotated benchmark dataset for multi sensor person tracking from a moving robot platform in a guide scenario, which will be made publicly available.

In tumor therapy, estimating tumor growth is crucial to get an early information regarding tumor therapy response and, if neccessary, adapt therapy. We propose a novel deep learning based algorithm using deep convolutional sparse autoencoders to find a minimal representation of tumor shape and texture for colorectal liver metastases. Furthermore, we provide a prediction of future lesion growth based on single slice CT tumor images which prospectively can be used as a prognosis for physicians. The state of the art in tumor treatment assessment for solid tumors mainly uses tumor diameter in single CT slices as the treatment response criterion (RECIST). However, whereas the correlation between RECIST and final treatment outcome was shown to be significant, its effect size is still limited. With our approach we achieve a Matthews correlation coefficient of 52.0% in predicting tumor treatment response compared to 28.2% with radiologic assessment, as well as an AUC of 0.814 opposed to 0.698.

Situation Awareness is a prominent concept in the human factors community. It is used to analyze and eliminate common sources of human errors in complex tasks and has seen wide-spread use in many fields, such as aviation, health care or ergonomics. Humans who are situation aware are able to reliably generate competent performance, a skill that is also highly desired for other autonomous agents. Yet, the concept has seen only limited use in robotics. We attest this to a lack of clear definitions which would allow assessing an artificial agents capacity for Situation Awareness. Our major contribution is an application-agnostic definition of the terminology and the processes involved in acquiring Situation Awareness. By integrating our definitions into the perception-action-cycle we provide a connection to the agent's observable behavior. Our second major contribution is a way to estimate, whether an agent has lost Situation Awareness based on surprise. This measure can be used online and does not require explicit or implicit knowledge of the task. We evaluate our concept on a physical workspace built for abstract Human- Robot-Cooperation scenarios.

Neuromuscular retraining is an important part of facial paralysis rehabilitation. To date, few publications have addressed the development of automated systems that support facial training. Current approaches require external devices attached to the patient's face, lack quantitative feedback, and are constrained to one or two facial training exercises. We propose an automated camera-based training system that provides global and local feedback for 12 different facial training exercises. Based on extracted 3D facial features, the patient's performance is evaluated and quantitative feedback is derived. The description of the feedback estimation is supplemented by a detailed experimental evaluation of the 3D feature extraction.

This paper introduces a novel robot-based approach to the stroke rehabilitation scenario, in which a mobile robot companion accompanies stroke patients during their walking self-training. This assistance enables them to move freely in the clinic practicing both their mobility and spatial orientation skills. Based on a set of questions for systematic evaluating the autonomy and practicability of assistive robots and a three-stage approach in conducting function and user tests in the clinical setting, we present the results of user trials performed with N=30 stroke patients in a stroke rehabilitation center between 4/2015 and 3/2016. This allowed us to make an honest inventory of the strengths and weaknesses of the developed robot companion and its already achieved practicability for clinical use. The results of the user studies show that patients and fellow patients were very open-minded and accepted the robotic coach. The robot motivated them for independent training and leaving their room, despite severe consequences of stroke (lower limbs paralysis, speech/language problems, loss of orientation, depression), provided a very self-determined training regime, and encouraged them to expand the radius of their training in the clinic.

Road condition acquisition and assessment are the key to guarantee their permanent availability. In order to maintain a country's whole road network, millions of high-resolution images have to be analyzed annually. Currently, this requires cost and time excessive manual labor. We aim to automate this process to a high degree by applying deep neural networks. Such networks need a lot of data to be trained successfully, which are not publicly available at the moment. In this paper, we present the GAPs dataset, which is the first freely available pavement distress dataset of a size, large enough to train high-performing deep neural networks. It provides high quality images, recorded by a standardized process fulfilling German federal regulations, and detailed distress annotations. For the first time, this enables a fair comparison of research in this field. Furthermore, we present a first evaluation of the state of the art in pavement distress detection and an analysis of the effectiveness of state of the art regularization techniques on this dataset.

One important function in assistive robotics for home applications is the detection of emergency cases, like falls. In this paper, we present a new detection system which can run on a mobile robot to detect persons after a fall event robustly. The system is based on 3D Normal Distributions Transform (NDT) maps on which a powerful segmentation is applied. Segments most likely belonging to a person lying on the ground are grouped into clusters. After extracting features with a soft encoding approach, each cluster is classified separately. Our experiments show that the system is able to reliably detect fallen persons in real-time. It clearly outperforms other 3D state-of-the-art approaches. We can show that our system is able to handle even very challenging situations, where fallen persons are very close to other objects in the apartment. Such complex fall events often occur in real-world applications.

During social interaction between humans and robots, body language can contribute to communicate mutual emotional states. In this paper, a method is presented, that enables wheeled robots with differential drive to express various emotional states by means of different movement styles during goal-directed motion. The motion control of the robot utilizes an objective-based motion planner optimizing the control commands in a high-dimensional search space by means of an evolutionary algorithm. The main contribution in this paper is an objective function that uses human feedback on the perceived robot emotion in order to evaluate the possible robot control sequences. First group experiments have been conducted in order to demonstrate the system and find suitable movement styles for possible emotional states.

We propose a real-time algorithm for the generic classification of humans and objects in 3D scenes. The algorithm does not depend on color information and works with depth data alone, making it very flexible for a wide area of applications. Further, we will show that it is very resistant to occlusion and will give correct classification results even in cases, where only a fraction of a full human or object can be captured by the depth sensor. Opposed to current approaches based on deep networks, training the IRON-BAG classifier (a bag-of-words model for IRON-features) can be done within minutes, making it easier to add new object classes, to finetune parameters and to adapt it to new operational scenarios. The system is easy to use, as it does not impose any constraints on the objects to detect, e.g. there's no limitation regarding shape, height, orientation, or position of humans and objects - knowledge of the sensor-pose or ground plane is not required. Instead of using depth images or point clouds as inputs for our classification pipeline, we solely operate on the NormalDistribution-Transform-map (NDT-map) data structure. NDT-maps provide a highly memory-efficient representation of depth data, and we show that the information contained within them is sufficient to accurately classify humans and objects from real-world 3D scenes with a speed of around 180 classifications per second on a single CPU core.

Robust person detection is required by many computer vision applications. We present a deep learning approach, that combines three Convolutional Neural Networks to detect people at different scales, which is the first time that a multi-resolution model is combined with deep learning techniques in the pedestrian detection domain. The networks learn features from raw pixel information, which is also rare for pedestrian detection. Due to the use of multiple Convolutional Neural Networks at different scales, the learned features are specific for far, medium, and near scales respectively, and thus, the overall performance is improved. Furthermore, we show, that neural approaches can also be applied successfully for the remaining processing steps of classification and non-maximum suppression. The evaluation on the most popular Caltech pedestrian detection benchmark shows that the proposed method can compete with state of the art methods without using Caltech training data and without fine tuning. Therefore, it is shown that our method generalizes well on domains it is not trained on.

Robust appearance-based person re-identification can only be achieved by combining multiple diverse features describing the subject. Since individual features perform different, it is not trivial to combine them. Often this problem is bypassed by concatenating all feature vectors and learning a distance metric for the combined feature vector. However, to perform well, metric learning approaches need many training samples which are not available in most real-world applications. In contrast, in our approach we perform score-level fusion to combine the matching scores of different features. To evaluate which score-level fusion techniques perform best for appearance-based person re-identification, we examine several score normalization and feature weighting approaches employing the the widely used and very challenging VIPeR dataset. Experiments show that in fusing a large ensemble of features, the proposed score-level fusion approach outperforms linear metric learning approaches which fuse at feature-level. Furthermore, a combination of linear metric learning and score-level fusion even outperforms the currently best non-linear kernel-based metric learning approaches, regarding both accuracy and computation time.

Rehabilitative follow-up care is important for stroke patients to regain their motor and cognitive skills. We aim to develop a robotic rehabilitation assistant for walking exercises in late stages of rehabilitation. The robotic rehab assistant is to accompany inpatients during their self-training, practicing both mobility and spatial orientation skills. To hold contact to the patient, even after temporally full occlusions, robust user re-identification is essential. Therefore, we implemented a person re-identification module that continuously re-identifies the patient, using only few amount of the robot's processing resources. It is robust to varying illumination and occlusions. State-of-the-art performance is confirmed on a standard benchmark dataset, as well as on a recorded scenario-specific dataset. Additionally, the benefit of using a visual re-identification component is verified by live-tests with the robot in a stroke rehab clinic.

This paper presents the implementation and evaluation results of the German research project SERROGA (2012 till mid 2015), which aimed at developing a robot companion for domestic health assistance for older people that helps keeping them physically and mentally fit to remain living independently in their own homes for as long as possible. The paper gives an overview of the developed companion robot, its system architecture, and essential skills, behaviors, and services required for a robotic health assistant. Moreover, it presents a new approach allowing a quantitative description and assessment of the navigation complexity of apartments to make them objectively comparable for function tests under real-life conditions. Based on this approach, the results of function tests executed in 12 apartments of project staff and seniors are described. Furthermore, the paper presents findings of a case study conducted with nine seniors (aged 68-92) in their own homes, investigating both instrumental and social-emotional functions of a robotic health assistant. The robot accompanied the seniors in their homes for up to three days assisting with tasks of their daily schedule and health care, without any supervising person being present on-site. Results revealed that the seniors appreciated the robot's health-related instrumental functions and even built emotional bonds with it.

This work introduces the IRON keypoint detector and the IRON descriptor which enable high-speed and high-accuracy alignment of 3D depth maps. Instead of using raw point values for storing 3D-scenes, all algorithms were designed to operate on Normal Distribution Transforms (NDT), since NDT-maps provide a highly memory-efficient representation of depth data. By taking into account surface curvature and object shape within NDT-maps, patches with strong surface variability can be recognized and described precisely. In this paper, the whole feature extraction process, as well as descriptor matching, outlier detection, and the final transform calculation between NDT-maps is elaborated. The presented technique is particularly insensitive to an initial offset between both maps, has a high robustness, and it achieves more than 75 NDT-map alignments per second (including complete memory allocation each time as well) in two large publicly available depth datasets while using only a single core of a modern Intel i7 CPU. Even though the main focus of this work was placed on the proposed IRON registration algorithm, two specific applications of this NDT-matching approach are outlined in the second part, namely robot pose tracking and NDT-one-shot localization within densely furnished domestic environments.

Intersections are the most accident-prone spots in the road network. In order to assist the driver in complex urban intersection situations, an ADAS will be required not only to recognize current but also to anticipate future maneuvers of the involved road users. Current approaches for intention estimation focus mainly on discerning only two intentions based on a vehicle's behavior. We argue that for distinguishing between more than two intentions not just a vehicle's kinematic behavior but also its driving situation needs to be taken into account. In our system we estimate four different intentions by modeling and recognizing driving situations in a Bayesian Network and using the behavior as additional evidence. For the behavior based estimation we present a newly engineered feature, the Anticipated Velocity at Stop line, that turned out to be a very strong indicator for the intention. Our system is evaluated on a real-world data set comprising approaches to seven different intersections on which we can show that our approach is able to estimate a driver's intention with a high accuracy.

In this paper, a dialog modeling approach for long-term interaction between a service robot and a single user is presented, which enables a user-adaptive interaction behavior of the robot. Central element of the dialog system is a probabilistic model of the user's reactions to the robot's behavior, which is learned online and used for a probabilistic planning process based on message passing in a dynamic factor graph. The suggested approach has been applied to implement a complex application on a mobile service robot, which has been tested in a 10 day evaluation study with 16 users in order to get a feedback on usability of the interaction design, adaptation skills, and feasibility of a rapid application development. Results and findings of that study are presented here briefly.

Non-contact image photoplethysmography has gained a lot of attention during the last 5 years. Starting with the work of Verkruysse et al. [1], various methods for estimation of the human pulse rate from video sequences of the face under ambient illumination have been presented. Applied on a mobile service robot aimed to motivate elderly users for physical exercises, the pulse rate can be a valuable information in order to adapt to the users conditions. For this paper, a typical processing pipeline was implemented on a mobile robot, and a detailed comparison of methods for face segmentation was conducted, which is the key factor for robust pulse rate extraction even, if the subject is moving. A benchmark data set is introduced focusing on the amount of motion of the head during the measurement.

People detection in 2D laser range data is a popular cue for person tracking in mobile robotics. Many approaches are designed to detect pairs of legs. These approaches perform well in many public environments. However, we are working on an assistance robot for stroke patients in a rehabilitation center, where most of the people need walking aids. These tools occlude or touch the legs of the patients. Thereby, approaches based on pure leg detection fail. The essential contribution of this paper are generic distance-invariant range scan features for people detection in 2D laser range data and the distinction of their walking aids. With these features we trained classifiers for detecting people without walking aids (or with crutches), people with walkers, and people in wheelchairs. Using this approach for people detection, we achieve an F1score of 0.99 for people with and without walking aids, and 86% of detections are classified correctly regarding their walking aid. For comparison, using state-of-the-art features of Arras et al. on the same data results in an F1score of 0.86 and 57% correct discrimination of walking aids. The proposed detection algorithm takes around 2.5% of the resources of a 2.8 GHz CPU core to process 270° laser range data at an update rate of 10 Hz.

As report on work in progress, this paper describes the objectives and the current state of implementation of the ongoing research project ROREAS (Robotic Rehabilitation Assistant for Stroke Patients), which aims at developing a robotic rehabilitation assistant for walking and orientation exercising in self-training during clinical stroke follow-up care. This requires strongly user-centered, polite and attentive social navigation and interaction behaviors that can motivate the patients to start, continue, and regularly repeat their self-training. Against this background, the paper gives an overview of the constraints and requirements arising from the rehabilitation scenario and the operational environment, a heavily populated multi-level rehabilitation center, and presents the robot platform ROREAS which is currently used for developing the demonstrators (walking coach and orientation coach). Moreover, it gives an overview of the robot's functional system architecture and presents selected advanced navigation and HRI functionalities required for a personal robotic trainer that can successfully operate in such a challenging real-world environment, up to the results of ongoing functionality tests and upcoming user studies.

The rising need for security in the last years has led to an increased use of surveillance cameras in both public and private areas. The increasing amount of footage makes it necessary to assist human operators with automated systems to monitor and analyze the video data in reasonable time. In this paper we summarize our work of the past three years in the field of intelligent and automated surveillance. Our proposed system extends the common active monitoring of camera footage into an intelligent automated investigative person-search and walk path reconstruction of a selected person within hours of image data. Our system is evaluated and tested under life-like conditions in real-world surveillance scenarios. Our experiments show that with our system an operator can reconstruct a case in a fraction of time, compared to manually searching the recorded data.

This paper presents results of user evaluations with a socially assistive robot companion for older people suffering from mild cognitive impairment (MCI) and living (alone) at home. Within the European FP7 project “CompanionAble” (2008-2012) [1], we developed assistive technologies combining a mobile robot and smart environment with the aim to support these people and assist them living in their familiar home environment. For a final evaluation, user experience studies were conducted with volunteer users who were invited to a test home where they lived and freely used the robot and integrated system over a period of two days. Services provided by the companion robot include reminders of appointments (pre-defined or added by the users themselves or their informal carer) as well as frequent recommendations to specific activities, which were listed e.g. by their family carers. Furthermore, video contact with relatives and friends, a cognitive stimulation game designed especially to counter the progress of cognitive impairments, and the possibility to store personal items with the robot are offered. Recognition of the user entering or leaving the home is triggering situation specific reminders like agenda items due during the (expected) absence, missed calls or items not to be forgotten. Continuing our previous work published in [2], this paper presents detailed description of the implemented assistive functions and results of user studies conducted during April and May 2012 in the smart house of the Dutch project partner Smart Homes in Eindhoven, The Netherlands.

In order to act socially compliant with humans, mobile robots need to show several behaviors that require the prediction of people's motion. For example, when a robot avoids a person, it needs to respect the human's personal space [1] and the avoidance behavior needs to be smooth, so that it is understandable to the interaction partner. To achieve this, the robot needs to reason about future paths a person is likely to follow. Because humans adapt their avoidance behavior to the robot's motion, the proposed method performs lifelong learning of the people's behavior while it adapts its own behavior to their motion. The human avoidance behavior is modeled by a discrete, multi-modal, spatio-temporal distribution over the people's future occurrences. This prediction is based on the people's positions and their velocities relatively to the robot and the obstacle situation of the robot's environment. The proposed prediction method is significantly better than a simple linear prediction. Particularly, for tactical decisions, like whether to avoid a moving person on the left or on the right side, this approach is well suited. Furthermore, when the humans get used to a robot, also a long-term change of the human behavior towards the robot can be learned by our approach.

Intersections are among the most complex traffic situations that motorists encounter, which is reflected by the fact that in Europe more than 40 percent of accidents resulting in injury occur at intersections. In order to support the driver in crossing an intersection an advanced driver assistance system is required to predict the behavior of other drivers, like acceleration and braking maneuvers, as accurately as possible. Such a prediction is a challenging task when considering the complexity and variability of situations encountered at urban intersections. We propose to tackle this problem using a two-staged approach. In the first stage the situation is decomposed into small, more manageable sets of related road users to prevent a combinatorial explosion of possibilities. For each set the road user's driving situation is estimated. In the second stage the velocity profiles of all road users are predicted, taking advantage of the previously estimated driving situation by employing prediction models that are specific to the situation type. The proposed method is evaluated on a simulated intersection situation where the two-staged approach clearly outperforms prediction methods that work without assessing driving situations first. We also show qualitative results on real-world data that confirm the benefits of our approach.

Mobile companion robots for elderly people are subject to recent research efforts. To provide useful services the robot must be aware of the position of the user. Since the field-of-view of the robot is limited, the user can easily leave the robot's perception, e.g. by going to another room. This paper proposes a method to search and locate a person not in the vicinity of the robot by driving through the apartment and checking for people. We apply restrictive as well as computationally expensive detection methods on-demand to verify the hypotheses of a real time person tracker. Additionally, the search-tour of the robot is guided by an occurrence probability histogram of previous positions of the user. The system has been tested by evaluating accuracy and time the robot needs to find a user in a 3-room scenario.

Falling down and not managing to get up again is one of the main concerns of elderly people living alone in their home. Robotic assistance for the elderly promises to have a great potential of detecting these critical situations and calling for help. This paper presents a feature-based method to detect fallen people on the ground by a mobile robot equipped with a Kinect sensor. Point clouds are segmented, layered and classified to detect fallen people, even under occlusions by parts of their body or furniture. Different features, originally from pedestrian and object detection in depth data, and different classifiers are evaluated. Evaluation was done using data of 12 people lying on the floor. Negative samples were collected from objects similar to persons, two tall dogs, and five real apartments of elderly people. The best feature-classifier combination is selected to built a robust system to detect fallen people.

Developing methods for people tracking on mobile robots is of great interest to engineers and scientists alike. Plenty of research is focused on pedestrian tracking in public areas. Yet, fewer work exists on practical people tracking in home environments with non-static cameras. This paper presents a real time people tracking system for mobile robots that filters asynchronous, multi-modal detections using a Kalman filter for each person. It allows for upright and sitting pose people tracking in home environments. We evaluate the performance of the tracking system using different detection modalities and compared it to state-of-the-art people detection methods. Evaluation was done on a newly collected indoor data set which we made publicly available for comparison and benchmarking.

In the field of human-robot interaction (HRI), detection, tracking and re-identification of humans in a robot's surroundings are crucial tasks, e. g. for socially compliant robot navigation. Besides the 3D position detection, the estimation of a person's upper-body orientation based on monocular camera images is a challenging problem on a mobile platform. To obtain real-time position tracking as well as upper-body orientation estimations, the proposed system comprises discriminative detectors whose hypotheses are tracked by a Kalman filter-based multi-hypotheses tracker. For appearance-based person recognition, a generative approach, based on a 3D shape model, is used to refine these tracked hypotheses. This model evaluates edges and color-based discrimination from the background. Furthermore, for each person the texture of his or her upper-body is learned and used for person re-identification. When computational resources are limited, the update rate of the model-based optimization reduces itself automatically. Thereby the estimation accuracy decreases, but the system keeps tracking the persons around the robot in real-time. The person's 3D pose is tracked up to a distance of 5.0 meters with an average Euclidean error of 18 cm. The achieved motion independent average upper-body orientation error is 22°. Furthermore, the upper-body texture is learned on-line which allowed a stable person re-identification in our experiments.

Fast and robust person reidentification is an important task in multi-camera surveillance and automated access control. We present an efficient appearance-based algorithm, able to reidentify a person regardless of occlusions, distance to the camera, and changes in view and lighting. The use of fast online feature selection techniques enables us to perform reidentification in hyper-real-time for a multi-camera system, by taking only 10 seconds for evaluating 100 minutes of HD-video data. We demonstrate, that our approach surpasses current appearance-based state-of-the-art in reidentification quality and computational speed and sets a new reference in non-biometric reidentification.

Laser based detection and tracking of persons can be used for numerous tasks, like statistical measurements for determining bottlenecks in public buildings, optimizing passenger flow, or planning camera placement. Only a network of multiple LRF is sufficient to fulfill these tasks in larger spaces. Calibrating multiple LRF into a global coordinate system is usually done by hand in a time consuming procedure. In this paper, we address the problem of automatically calibrating such a sensor network. We introduce an automatic calibration mechanism, which is able to obtain the positions and orientations of all LRF in a global coordinate system, without any prior knowledge of the scene. Our approach is based on comparing person tracks, determined by each individual LRF unit and matching them in order to obtain constraints between the LRF units. By resolving these constraints, we are able to estimate the poses of all LRF. We evaluate and compare our method to the current state of the art approach methodically and experimentally. Experiments show that our calibration approach outperforms this approach.

In this paper, we present MIRA, a new middleware for robotic applications. It is designed for use in real-world applications and for research and teaching. In comparison to many other existing middlewares, MIRA employs novel techniques for communication that are described in this paper. Moreover, we present benchmarks that analyze the performance of the most commonly used middlewares ROS, Yarp, LCM, Player, Urbi, and MOOS. Using these benchmarks, we can show that MIRA outperforms the other middlewares in terms of latency and computation time.

Usually, in mobile robotics the robot has to deal with tasks like interacting with a person or performing a driving task. But what happens, if the robot just has to wait and thereby still has to react on user commands? In this case, the robot has to find a good position where the user can still be observed, and the robot does not disturb the user's activities. Such a position has to fulfill different criteria: first, to guarantee the observability of the person with the robots on-board sensors and second, the robot should be able to observe the person when the person changes its position at its resting place. In this paper, a new approach is presented how to find a position, providing all these aspects, by solving an optimization problem using a particle swarm optimizer. We also present first results for that problem in the 2D and 3D case.

Fast vision-based object and person tracking is important for various applications in mobile robotics and Human-Robot Interaction. While current state-of-the-art methods use descriptive features for visual tracking, we propose a novel approach using a sparse template based feature set, which is drawn from homogeneous regions on the object to be tracked. Using only a small number of simple features, without complex descriptors in combination with logarithmic-search, the tracker performs at hyper-real-time on HD-images without the use of parallelized hardware. Detailed benchmark experiments show that it outperforms most other state-of-the-art approaches for real-time object and person tracking in quality and runtime. In the experiments we also show the robustness of the tracker and evaluate the effects of different initialization methods, feature sets, and parameters on the tracker. Although we focus on the scenario of person and object tracking in robot applications, the proposed tracker can be used for a variety of other tracking tasks.

Laser based detection and tracking of persons can be used for numerous tasks. While a single laser range finder (LRF) is sufficient for detecting and tracking persons on a mobile robot platform, a network of multiple LRF is required to observe persons in larger spaces. Calibrating multiple LRF into a global coordinate system is usually done by hand in a time consuming procedure. An automatic calibration mechanism for such a sensor network is introduced in this paper. Without the need of prior knowledge about the environment, this mechanism is able to obtain the positions and orientations of all LRF in a global coordinate system. By comparing person tracks, determined for each individual LRF unit and matching them, constrains between the LRF units can be calculated. We are able to estimate the poses of all LRF by resolving these constrains. We evaluate and compare our method to the current state of the art approach methodically and experimentally. Experiments show that our calibration approach outperforms this approach.

In this paper, we present a monocular, texture-based method for person detection and upper-body orientation classification. We build on a commonly used approach for person recognition that uses a Support Vector Machine (SVM) on Histograms of Oriented Gradients (HOG) [1] but replace the SVM by a decision tree with SVMs as binary decision makers. Thereby, in addition to the pure detection of persons, the distinction of eight upper-body orientation classes is enabled. The detection of humans and the estimation of their upper-body orientation from larger distances is essential for socially acceptable navigation of mobile robots. It permits to estimate the human's notice of the robot or even the human's interest in an interaction. Thus, it is the basis for the decision whether to approach or to avoid a human. By using an SVM decision tree for upper-body orientation estimation in discrete steps of 45°, we were able to classify about 64% of the test samples with an absolute error of less than 22.5°. This performance is much better than the results we obtained with comparable methods. Furthermore, our approach proved to be faster than the other state-of-the-art methods. This is of high relevance for implementation on mobile robots with limited computational resources.

This paper presents an architectural overview of a robot-based visitor information system in a university building. Two mobile robots serve as mobile information terminals providing information about the employees, labs, meeting rooms, and offices in the building and are able to guide the visitors to these points of interest. The paper focuses on the different software components needed to meet the requirements of the multi-story office building. Furthermore, the integration of a multi-hypotheses person tracker is outlined, which helps the robots to interact with the people in their near surrounding. Besides first observations on interaction, the further development is outlined as well.

This paper presents results of the development of a socially assistive home robot companion for older people suffering from mild cognitive impairment (MCI) and living (alone) at home. This work was part of the European FP7 project “CompanionAble” (2008-2012) [1] which aimed at developing assistive technologies that can support these elderly and help them to remain in their familiar living environment for as long as possible. To overcome current market entry barriers, from the start we consistently adopted a user- and application-centered development process of the companion robot and focused on three main aspects: (i) the realization of a set of mandatory functionalities to support care recipients and caregivers, (ii) a strict design and usability driven realization to increase the acceptance of the robot by the different end-user groups (the elderly, their relatives, and caregivers), and (iii) the development and component selection considering production and operational costs. In continuation of the work presented in [2], this paper describes the final implementation of the companion robot and presents latest results of functional tests and early findings of user studies recently conducted in the smart house of the Dutch project partner Smart Homes in Eindhoven, The Netherlands.

For robot mapping occupancy grid maps are the most common representation of the environment. However, most existing algorithms for creating such maps assume a fixed resolution of the grid cells. In this paper we present a novel mapping technique that chooses the resolution of each cell adaptively by merging and splitting cells depending on the measurements. The splitting of the cells is based on a statistical measure that we derive in this paper. In contrast to other approaches the adaption of the resolution is done online during the mapping process itself. Additionally, we introduce the Nd-Tree, a generalization of quadtrees and octrees that allows to subdivide any d-dimensional volume recursively with Ndchildren per node. Using this data structure our approach can be implemented in a very generic way and allows the creation of 2D, 3D and even higher dimensional maps using the same algorithm. Finally, we show results of our proposed method for 2D and 3D mapping using different kinds of range sensors.

The paper is addressing several aspects of our work as part of the European FP7 project ¿CompanionAble¿ and gives an overview of the progress in developing a socially assistive home robot companion for elderly people with mild cognitive impairment (MCI) living alone at home. The spectrum of required assistive functionalities and services that have been specified by the different end-user target groups of such a robot companion (the elderly, relatives, caregivers) is manifold. It reaches from situation-specific, intelligent reminding (e.g. taking medication or drinking) and cognitive stimulation, via mobile videophony with relatives or caregivers, up to the autonomous detection of dangerous situations, like falls, and their evaluation by authorized persons via mobile telepresence. From the beginning, our approach has been focused on long-term and everyday suitability and low-cost producibility as important prerequisites for the marketability of the robot companion. Against this background, the paper presents the main system requirements derived from user studies, the consequences for the hardware design and functionality of the robot companion, its system architecture, a key technology for HRI in home environments - the autonomous user tracking and searching, up to the results of already conducted and ongoing functionality tests and upcoming user studies.

The paper gives an overview of the progress in developing a socially assistive home robot companion for elderly people with mild cognitive impairment (MCI) living alone at home. The spectrum of required assistive functionalities of such a robot companion is broad and reaches from reminding functions (e.g. taking medication or drinking) and cognitive stimulation exercises, via mobile videophony with relatives or caregivers, up to the detection and evaluation of critical situations, like falls. The paper is addressing several aspects of our work as part of the European FP7 project “CompanionAble”, as for example the developed robot hardware and its software and control architecture, the implemented skills for robust user detection and tracking and user-centered navigation in the home environment, and reports on already conducted and still ongoing functionality testings and pending usability studies with the end-user target groups (the elderly, relatives, caregivers).

Robust and reliable obstacle detection is an important capability for mobile robots. In our previous works we have presented an approach for visual obstacle detection based on feature based monocular scene-reconstruction. Most existing feature-based approaches for visual SLAM and scene reconstruction select their features uniformly over the whole image based on visual saliency only. In this paper we present a novel attention-driven approach that guides the feature selection to image areas that provide the most information for mapping and obstacle detection. Therefore, we present an information theoretic derivation of the expected information gain that results from the selection of new image features. Additionally, we present a method for building a volumetric representation of the robots environment in terms of an occupancy voxel map. The voxel map provides top-down information that is needed for computing the expected information gain. We show that our approach for guided feature selection improves the quality of the created voxel maps and improves the obstacle detection by reducing the risk of missing obstacles.

This paper presents a model based system for a mobile robot to find an optimal pose for the observation of a person in indoor living environments. We define the observation pose as a combination of the camera position and view direction as well as further parameters like the aperture angle. The optimal placement of a camera is not trivial because of the high dynamic range of the scenes near windows or other bright light sources, which often results in poor image quality due to glare or hard shadows. The proposed method tries to minimize these negative effects by determining an optimal camera pose based on two major models: A spatial free space model and a representation of the lighting. In particular, a task-dependent optimization takes into account the intended purpose of the camera images, e.g. different inputs are needed for video communication with other people or for an image-processing based passive observation of the person's activities. To prove the validity of our approach, we present first experimental results comparing the chosen observation pose and resulting image with and without respect to lighting in different observation tasks.

The paper gives a comprehensive overview of our Shopping Guide project, which aims at the development of interactive mobile shopping companion robots for everyday use in challenging operating environments such as home improvement stores. It is spanning an arc from the expectations and requirements of store owners and customers, via the challenges of the shopping scenario and the operating environment, the implemented functionality of the shopping guide robots, up to the results of long-term field trials. The field trials started in April 2008 and still ongoing aim at studying whether and how a group of interactive mobile shopping guide robots can operate completely autonomously in such everyday environments and how they are accepted by uninstructed customers. In these field trials, where nine robotic shopping guides traveled together 2187 kilometers in three different home improvement stores in Germany, more than 8,600 customers were successfully guided to the locations of their products of choice. With the successful development of these shopping guide robots, a further important step towards assistive robotics for daily use has been done.

In this paper, we present an approach for facial expression classification, based on Active Appearance Models. To be able to work in real-world, we applied the AAM framework on edge images, instead of gray images. This yields to more robustness against varying lighting conditions. Additionally, three different facial expression classifiers (AAM classifier set, MLP and SVM) are compared with each other. An essential advantage of the developed system is, that it is able to work in real-time - a prerequisite for the envisaged implementation on an interactive social robot. The real-time capability was achieved by a two-stage hierarchical AAM tracker and a very efficient implementation.

In continuation of our previous work on visual, appearance-based localization in manually built maps in this paper we present a novel appearance-based, visual SLAM approach. The essential contribution of this work is, an adaptive sensor model which is estimated online and a graph matching scheme to evaluate the likelihood of a given topological map. Both methods enable the combination of an appearance-based, visual localization concept with a Rao-Blackwellized Particle Filter (RBPF) as state estimator to a real-world suitable, online SLAM approach. In our system, each RBPF particle incrementally constructs its own graph-based environment model which is labeled with visual appearance features (extracted from panoramic 360deg snapshots of the environment) and the estimated poses of the places where the snapshots were captured. The essential advantages of this appearance-based SLAM approach are its low memory and computing-time requirements. Therefore, the algorithm is able to perform in real-time. Finally, we present the results of SLAM experiments in two challenging environments that investigate the stability and localization accuracy of this SLAM technique.

In this paper, we present the application of our generic, sensor-independent Map Match SLAM framework to visual mapping. In our previous work , we have introduced the map match SLAM approach for mapping with sonar range readings: Extending the grid-based Rao-Blackwellized particle filter SLAM approach, in Map Match SLAM, a local map is maintained by each particle in addition to the global map. The local map is used to represent the most recent observations, and weighting of the particles is done based on the compliance of the local and the global map. In this paper, we show how RBPF SLAM can also be applied for mapping and path reconstruction with a stereo camera or a single monocular camera, respectively. By mapping with completely different sensors such as sonar, stereo, or monocular cameras, we prove the wide range applicability of RBPF SLAM and our map match SLAM computational framework.

Correctly identifying people who are interested in an interaction with a mobile robot is an essential task for a smart Human-Robot Interaction. In this paper an approach is presented for selecting suitable trajectory features in a task specific manner from a huge amount of different forms of possible representations. Different sub-sampling techniques are proposed to generate trajectory sequences from which features are extracted. The trajectory data was generated in real world experiments that include extensive user interviews to acquire information about user behaviors and intentions. Using those feature vectors in a classification method enables the robot to estimate the user's interaction interest. For generating low-dimensional feature vectors, a common method, the Principle Component Analysis, is applied. The selection and combination of useful features out of a set of possible features is carried out by an information theoretic approach based on the Mutual Information and Joint Mutual Information with respect to the user's interaction interest. The introduced procedure is evaluated with neural classifiers, which are trained with the extracted features of the trajectories and the user behavior gained by observation as well as user interviewing. The results achieved indicate that an estimation of the user's interaction interest using trajectory information is feasible.

The paper describes progress achieved in our long-term research project ShopBot, which aims at the development of an intelligent and interactive mobile shopping assistant for everyday use in shopping centers or home improvement stores. It is focusing on recent progress concerning two important methodological aspects: (i) the on-line building of maps of the operation area by means of advanced Rao-Blackwellized SLAM approaches using both sonar-based gridmaps as well as vision-based graph maps as representations, and (ii) a probabilistic approach to multi-modal user detection and tracking during the guidance tour. Experimental results of both the map building characteristics and the person tracking behavior achieved in an ordinary home improvement store demonstrate the reliability of both approaches. Moreover, we present first very encouraging results of long-term field trials which have been executed with three robotic shopping assistants in another home improvement store in Bavaria since March 2008. In this field test, the robots could demonstrate their suitability for this challenging real-world application, as well as the necessary user acceptance.

We propose a method for the joint estimation of formant trajectories from spectrograms. Formants are enhanced in the spectrograms obtained from the application of a Gammatone filterbank via a smoothing along the frequency axis. In contrast to previously published approaches, the used tracking algorithm relies on the joint distribution of formants rather than using independent tracker instances. More precisely, Bayesian mixture filtering in conjunction with adaptive frequency range segmentation as well as Bayesian smoothing are used. The algorithm was evaluated on a publicly available database containing hand-labeled formant tracks. Experimental results show a significant performance improvement compared to a state of the art approach.

Speech has become an important part in human robot interaction (HRI), e.g. for person detection systems by using localized sound sources or for applications in automatic speech recognition (ASR) systems. By using speech in HRI in real world environments, we have to deal with mostly high and varying background noise, reverberation and also with different sound sources superimposing speech and other noises. Therefore, for real world scenarios a suitable signal preprocessing is essential. In this paper, we present a part of the artificial auditory system implemented on the mobile interaction robot HOROS using only two low cost microphones. We combined neural voice activity detection (VAD) and adaptive noise reduction which are essential aspects for HRI using mobile robot systems in changing and populated real-world environments. In the result, our system is able to robustly react on speech signals from its human interaction partner while ignoring other sound sources. Experiments show a significantly improved ASR performance in demanding environments making the system suitable for the use in real-world scenarios.

In this paper, we present two important aspects of our human-robot communication interface which is being developed in the context of our long-term research framework PERSES dealing with the development of highly interactive mobile robotic assistants. First, we introduce a multi-modal people detection and tracking system, a fundamental prerequisite for the observation of a human interaction partner and his nonverbal instructions given by pointing poses, gestures, head pose and eye gaze. Based on this detection and tracking system, we present a hierarchical neural architecture that is capable of estimating a target point at the floor given a pointing pose, thus enabling a user to command his mobile robot to a specific target position in his local surroundings by means of pointing. In this context, we were especially interested in determining whether it is possible to accomplish such a target point estimator using only monocular images of low-cost cameras. Both the tracker and the target point estimator were implemented and experimentally investigated on our mobile robotic assistant HOROS. The achieved recognition results presented finally demonstrate that it is in fact possible to realize a user-independent pointing pose estimation using monocular images only, but further efforts are necessary to improve the robustness of this approach for everyday application.

In this paper, we present a neural architecture that is capable of estimating a target point from a pointing gesture, thus enabling a user to command a mobile robot to a specific position in his local surroundings by means of pointing. In this context, we were especially interested to determine whether it is possible to implement a target point estimator using only monocular images of low-cost Webcams. The feature extraction is also quite straightforward: We use a gabor jet to extract the feature vector from the normalized camera images; and a cascade of multi layer perceptron (MLP) classifiers as estimator. The system was implemented and tested on our mobile robotic assistant HOROS. The results indicate that it is in fact possible to realize a pointing estimator using monocular image data, but further efforts are necessary to improve the accuracy and robustness of our approach

For socially interactive robots it is essential to be able to estimate the interest of people to interact with them. Based on this estimation the robot can adapt its dialog strategy to the different people's behaviors. Consequently, efficient and robust techniques for people detection and tracking are basic prerequisites when dealing with human-robot interaction (HRI) in real-world scenarios. In this paper, we introduce an imposed approach for integration of several sensor modalities and present a multimodal, probability-based people detection and tracking system and its application using the different sensory systems of our mobile interaction robot HOROS. For each of these sensory cues, separate and specific Gaussian distributed hypotheses are generated and further merged into a robot-centered map by means of a flexible probabilistic aggregation scheme based on covariance intersection (CI). The main advantages of this approach are the simple extensibility by integration of further sensory channels, even with different update frequencies, and the usability in real-world HRI tasks. Finally, promising experimental results achieved for people tracking in a real-world environment, and university building, are presented

This paper describes a novel omnivision-based concurrent map-building and localization (CML) approach which is able to robustly localize a mobile robot in a uniformly structured, maze-like environment with changing appearances. The presented approach extends and improves known appearance-based CML techniques in a few essential aspects. For example, an advanced learning scheme in combination with an active forgetting is introduced to allow a complexity restricting adaptation of the environment model to appearance variations of the operation area. Moreover, a generalized scheme for fusion of localization hypotheses from several state estimators with different meaning and certainty and a distributed coding of the current observation by a weighted set of reference observations is proposed. Finally, several real-world localization experiments investigating the stability and localization accuracy of this novel omnivision-based CML technique for a highly dynamic and populated operation area, a home store, are presented.

We introduce a computational model of sensor fusion based on the topographic representations of a "two-microphone and one camera" configuration. Our aim is to perform a robust multimodal attention-mechanism in artificial systems. In our approach, we consider neurophysiological findings to discuss the biological plausibility of the coding and extraction of spatial features, but also meet the demands and constraints of applications in the field of human-robot interaction. In contrast to the common technique of processing different modalities separately and finally combine multiple localization hypotheses, we integrate auditory and visual data on an early level. This can be considered as focusing the attention or controlling the gaze onto salient objects. Our computational model is inspired by findings about the inferior colliculus in the auditory pathway and the visual and multimodal sections of the superior colliculus. Accordingly it includes: a) an auditory map, based on interaural time delays, b) a visual map, based on spatio-temporal intensity difference and c) a bimodal map where multisensory response enhancement is performed and motor-commands can be derived. After introducing a modified Amari-neural field architecture in the bimodal model, we place emphasis on a novel method of evaluation and parameter-optimization based on biology-inspired specifications and real-world experiments.

This paper describes a system developed for a mobile service robot which detects and tracks the position of a user's face in 3D-space using a vision (skin color) and a sonar based component. To make the skin color detection robust under varying illumination conditions, it is supplied with an automatic white balance algorithm. The hypothesis of the user's position is used to orient the robot's head towards the current user allowing it to grab high resolution images of his face suitable for verifying the hypothesis and for extracting additional information.

The basic idea of our omniview-based MCL approach and preliminary experimental results were presented in our previous paper [Proc. IROS 2002, pp. 256-262]. In continuing, this paper describes a number of methodical and technical improvements addressing challenges arising from the characteristics of our real-world application, the vision-based self-localization of a mobile robot that acts as a shopping assistant in the maze-like environment of a home store. To cope with highly variable illumination conditions, we present a reference-based correction approach that realizes a robust, automatic luminance stabilization and color adaptation already at the level of image formation. To deal with severe occlusions or disturbances of the omnidirectional image caused by, e.g. people standing near the robot or local illumination artifacts, we introduce a novel selective observation comparison method as prerequisite for a robust particle filter update. Further studies investigate the impact of the utilized observation model on the localization accuracy. The results of a series of localization experiments carried out in the home store confirm the robustness and superiority of our advanced, real-time approach.

We present a novel omnivision-based robot localization approach which utilizes the Monte Carlo Localization (MCL), a Bayesian filtering technique based on a density representation by means of particles. The capability of this method to approximate arbitrary likelihood densities is a crucial property for dealing with highly ambiguous localization hypotheses as are typical for real-world environments. We show how omnidirectional imaging can be combined with the MCL-algorithm to globally localize and track a mobile robot given a taught graph-based representation of the operation area. In contrast to other approaches, the nodes of our graph are labeled with both visual feature vectors extracted from the omnidirectional image, and odometric data about the pose of the robot at the moment of the node insertion (position and heading direction). To demonstrate the reliability of our approach, we present first experimental results in the context of a challenging robotics application, the self-localization of a mobile service robot acting as shopping assistant in a very regularly structured, maze-like and crowded environment, a home store.

Presents vision-based robot navigation and user localization techniques of our long-term research project PERSES (personal service system), which aims to develop an interactive mobile shopping assistant that allows a continuous and intuitively understandable interaction with customers in a home store. Against this background, the paper describes a number of new or improved approaches, addressing challenges arising from the characteristics of the operation area, and from the need to continuously interact with users in a complex environment. With our approaches to vision-based or visually-controlled map building, self-localization and navigation as well as user localization and tracking, we want to make a contribution to the real-world suitability of interactive mobile service-robots in non-trivial application areas and demanding human-robot interaction scenarios.

In this paper we present a biologically inspired neural architecture for visual perception based on anticipation. The main goal of this work is to demonstrate, that anticipation is a central key to improve the perception performance of technical systems. The presented approach is able to increase the robustness of the perception process against noise or sensory dropouts. We demonstrate these perceptional improvements through our architecture at the level of local navigation behavior of the miniature robot Khepera. We claim that perception is not an end in itself. Instead it is a sensorimotor process integrating the generation of behavior.

We present a new control scheme for an industrial hard-coal combustion process in a power plant based on reinforcement-learning in combination with neural networks. To comply with the great requirements for environmental protection, the plant operator is interested in a minimization of the nitrogen oxides emission and a maximization of the efficiency factor, while other process parameters have to be kept within predefined limits. To cope with both the tremendous action and state space of the power plant, we present a multiagent-reinforcement-system consisting of 4 agents, which are realized by relatively simple neural function approximators. We demonstrate that our multiagent-system was able to significantly reduce the overall air consumption of the real combustion process of the power plant.

Several systems for automatic gesture recognition have been developed using different strategies and approaches. In these systems the recognition engine is mainly based on three algorithms: dynamic pattern matching, statistical classification, and neural networks (NN). In this paper three architectures for the recognition of dynamic gestures using the above mentioned techniques or a hybrid combination of them are presented and compared. For all architectures a common preprocessor receives as input a sequence of color images, and produces as output a sequence of feature vectors of continuous parameters. The first two systems are hybrid architectures consisting of a combination of neural networks and hidden Markov models (HMM). NNs are used for the classification of single feature vectors while HMMs for the modeling of sequences of them with the aim to exploit the properties of both these tools. More precisely, in the first system a Kohonen feature map (SOM) clusters the input space. Further, each code-book is transformed into a symbol from a discrete alphabet and fed into a discrete HMM for classification. In the second approach a radial basis function (RBF) network is directly used to compute the HMM state observation probabilities. In the last system only dynamic programming techniques are employed. An input sequence of feature vectors is matched by some predefined templates by using the dynamic time warping (DTW) algorithm. Preliminary experiments with our baseline systems achieved a recognition accuracy up to 92%. All systems use input from a monocular color video camera, are user-independent but so far, they are not yet real-time.

We describe a biological inspired model of binaural sound localization using interaural time differences (ITDs). To handle the problem of temporal coding and to facilitate a hardware implementation in analog VLSI, the simulation of the system is based on a spike response model. This neuron model takes up physiological properties like postsynaptic potentials (PSPs) and a refractory period. A winner-take-all (WTA) network selects the dominant source from the representation of the sound's angles of incidences, and can be biased by a multisensory support. We use simulations on real audio data to investigate the function and the practical application of the system.

Several systems for automatic gesture recognition have been developed using different strategies and approaches. In these systems the recognition engine is mainly based on three algorithms: dynamic pattern matching, statistical classification, and neural networks (NN). In that paper we present four architectures for gesture-based interaction between a human being and an autonomous mobile robot using the above mentioned techniques or a hybrid combination of them. Each of our gesture recognition architecture consists of a preprocessor and a decoder. Three different hybrid stochastic/connectionist architectures are considered. A template matching problem by making use of dynamic programming techniques is dealt with; the strategy is to find the minimal distance between a continuous input feature sequence and the classes. Preliminary experiments with our baseline system achieved a recognition accuracy up to 92%. All systems use input from a monocular color video camera, and are user-independent but so far they are not in real-time yet.

We present a control scheme based on reinforcement learning for an industrial hard-coal combustion process in a power plant. To comply with the great demands on environmental protection, the plant operator is interested in a minimization of the nitrogen oxides emission, while other process parameters have to be kept within predefined limits. To cope with both the tremendous action and situation space of the power plant, we present a multiagent reinforcement system consisting of 4 agents, which are realized by relatively simple neural function approximators. We demonstrate, that our multiagent system was able to significantly reduce the overall air consumption of the real combustion process of the power plant.

In this paper, we present a hybrid neural architecture to predict optical flow fields as consequences of real and hypothetical actions. In this architecture, we introduce a neural field-based method to fuse sensory bottom-up and predicted top-down expectations. All subsystems extensively use confidence estimations to reduce disturbances caused by noise. The facilities of this anticipative preprocessing can be demonstrated by means of an optical flow field based local navigation behavior of the miniature robot KHEPERA. Our anticipative preprocessing enables the robot to bridge gaps of sensory dropouts and, in consequence, to avoid collisions even with very noisy sensory information.

The paper describes the general idea, the application scenario, and selected methodological approaches of our long term research project PERSES (PERsonal SErvice System). The aim of the project consists of the development of an interactive mobile shopping assistant that allows a continuous and intuitively understandable interaction with a customer in a home improvement store. Typical tasks we have to tackle are to detect and contact potential users in the operation area, to guide them to desired areas or articles within the store or to follow them as a mobile information kiosk while continuously observing their behavior. Due to the specificity of the interaction-oriented scenario and the characteristics of the operation area, we have focused on vision based methods for both human-robot interaction and robot navigation. Besides some methodological approaches, we present preliminary results of experiments achieved with our mobile robot PERSES in the store with an emphasis on vision based methods for user localization, map building and self-localization.

The basic idea of our approach is to avoid the separation of perception and the generation of action. Therefore, information extracted from the environment is selected according to the relevance of the system's intended action. Task-specific attention emerges in an attention selection map from a competition among several hypotheses in consideration of task relevant subgoals. This decision initiates the emergence of a task-specific focus of attention in a saliency map which refers to a textured or coloured region in a scene. The networks underlying the attention selection map and the saliency map consist of interacting columns with local excitatory and global inhibitory coupled feedback. In the attention selection map lateral cooperation is used as a way to integrate task pertinent subgoals. Implemented subgoals are the size of regions, the security of a classification hypothesis and the valuation of the hypothesis for the task. The performance is demonstrated on a real-world selection of textured objects for a robot grasping task.