Taking over arbitrary tasks like humans do with a mobile service robot in open-world settings requires a holistic scene perception for decision-making and high-level control. This article presents a human-inspired scene perception model to minimize the gap between human and robotic capabilities. The approach takes over fundamental neuroscience concepts, such as a triplet perception split into recognition, knowledge representation, and knowledge interpretation. A recognition system splits the background and foreground to integrate exchangeable image-based object detectors and simultaneous localization and mapping, a multilayer knowledge base represents scene information in a hierarchical structure and offers interfaces for high-level control, and knowledge interpretation methods deploy spatio-temporal scene analysis and perceptual learning for self-adjustment. A single-setting ablation study is used to evaluate the impact of each component on the overall performance for a fetch-and-carry scenario in two simulated and one real-world environment.

Robotic applications like automated order picking in warehouses or retail stores, or fetch and carry tasks in hospitals, care homes, or households rely on the capability of service robots to find and handle a specific type of object. These applications are challenging as the set of objects is very large and varies over time. Despite its significance, there is no suitable universal large-scale dataset available from the retail domain, which allows for a principled analysis of all relevant robotics research aspects in that field. Hence, this paper introduces a novel dataset of more than 1,000 retail objects, including color images, 3D scans, and high-resolution textured 3D models of individual objects, synthetic scenes and real settings, which covers the specifics of the retail domain. The dataset was designed to serve researchers in all relevant robotics tasks in retail like 3D reconstruction and object modeling, large-scale object classification and instance detection including incremental learning and fine-grained detection, text reading, logo detection, semantic grounding and affordance detection, grasp analysis and manipulation planning, as well as digital twinning and virtual environments. Based on synthetic RGB images of scenes created from the 3D models, two exemplary use cases are examined in this paper to demonstrate the benefits of the dataset: we evaluate the state-of-the-art incremental object detection method InstanceNet and a few-shot fine-grained object classification method. The results prove the suitability of InstanceNet for incremental object detection on large datasets and are promising for the few-shot object classification system.

This study presents key technologies of a mobile service robot developed to manipulate objects around people safely. We demonstrate this ability to support staff in elderly care homes in the future. The take-over by a service robot allows the staff to spend less time with routine logistical tasks and therefore better focus on the interaction with residents. In the selected application scenario, the robot helps staff by (1) retrieving empty bottles from the residents' rooms, (2) bringing them to the kitchen, (3) taking the refilled bottles back to a table inside the residents' rooms. This task seems trivial for a person, but the robot needs to orchestrate numerous algorithms and components to work smoothly, such as bottle pose detection, manipulation, and navigation. A technical evaluation indicates a high performance of single components, but due to isolated failures, the overall scenario does not always succeed. Next to the technical aspects, it is fundamental to determine the acceptance of the robot, which was achieved by analyzing questionnaires given to care workers. Finally, this paper presents lessons learned to help other researchers in similar use-cases.

One way to allow elderly people to stay longer in their homes is to use of service robots to support them with everyday tasks. Having this goal in mind, we design, develop, and evaluate a low-cost mobile robot to communicate with elderly people. The main idea is to create an affordable communication assistant robot which is optimized for multimodal Human-Robot Interaction (HRI). Our robot can navigate autonomously through dynamic environments using a new algorithm to calculate poses for approaching persons. The robot was tested in a real-life scenario in a residential care home for the elderly.