Robots operating in everyday environments encounter a wide variety of previously unseen objects. Deep Learning methods simplify unknown object and scene segmentation by structuring inherent real-world complexities, improving visual scene understanding. However, they need vast amounts of labeled high-variance data for training. Acquiring these labels for rich real-world data requires significant manual effort, especially for segmentation masks. Although interactive segmentation accelerates this process, these methods still require substantial manual interaction, and the creation of large datasets remains labor-intensive. Consequently, there is a lack of diverse, high-quality datasets for unknown object instance segmentation in everyday environments. This research proposes a semi-automatic, RGB-only algorithmic pipeline for annotating novel objects, reducing manual effort to iteratively placing objects in the scene. We investigate several change detection-based approaches, including remote sensing change detection methods (TTP model), the DeepBackgroundMattingV2 image matting model, and the Segment Anything Model (SAM1 + SAM2) prompted with automatically extracted change regions. We propose the novel ILIS dataset to evaluate these methods in challenging everyday scenes, displaying reliable automatic mask proposal performance of up to 0.9549 mIoU and 0.9565 boundary F1 score. This highlights the potential of this method to accelerate large-scale dataset creation, saving at least 27.27 hours per 1,000 images by eliminating manual annotations.

Taking over arbitrary tasks like humans do with a mobile service robot in open-world settings requires a holistic scene perception for decision-making and high-level control. This article presents a human-inspired scene perception model to minimize the gap between human and robotic capabilities. The approach takes over fundamental neuroscience concepts, such as a triplet perception split into recognition, knowledge representation, and knowledge interpretation. A recognition system splits the background and foreground to integrate exchangeable image-based object detectors and simultaneous localization and mapping, a multilayer knowledge base represents scene information in a hierarchical structure and offers interfaces for high-level control, and knowledge interpretation methods deploy spatio-temporal scene analysis and perceptual learning for self-adjustment. A single-setting ablation study is used to evaluate the impact of each component on the overall performance for a fetch-and-carry scenario in two simulated and one real-world environment.

Robotic applications like automated order picking in warehouses or retail stores, or fetch and carry tasks in hospitals, care homes, or households rely on the capability of service robots to find and handle a specific type of object. These applications are challenging as the set of objects is very large and varies over time. Despite its significance, there is no suitable universal large-scale dataset available from the retail domain, which allows for a principled analysis of all relevant robotics research aspects in that field. Hence, this paper introduces a novel dataset of more than 1,000 retail objects, including color images, 3D scans, and high-resolution textured 3D models of individual objects, synthetic scenes and real settings, which covers the specifics of the retail domain. The dataset was designed to serve researchers in all relevant robotics tasks in retail like 3D reconstruction and object modeling, large-scale object classification and instance detection including incremental learning and fine-grained detection, text reading, logo detection, semantic grounding and affordance detection, grasp analysis and manipulation planning, as well as digital twinning and virtual environments. Based on synthetic RGB images of scenes created from the 3D models, two exemplary use cases are examined in this paper to demonstrate the benefits of the dataset: we evaluate the state-of-the-art incremental object detection method InstanceNet and a few-shot fine-grained object classification method. The results prove the suitability of InstanceNet for incremental object detection on large datasets and are promising for the few-shot object classification system.

This study presents key technologies of a mobile service robot developed to manipulate objects around people safely. We demonstrate this ability to support staff in elderly care homes in the future. The take-over by a service robot allows the staff to spend less time with routine logistical tasks and therefore better focus on the interaction with residents. In the selected application scenario, the robot helps staff by (1) retrieving empty bottles from the residents' rooms, (2) bringing them to the kitchen, (3) taking the refilled bottles back to a table inside the residents' rooms. This task seems trivial for a person, but the robot needs to orchestrate numerous algorithms and components to work smoothly, such as bottle pose detection, manipulation, and navigation. A technical evaluation indicates a high performance of single components, but due to isolated failures, the overall scenario does not always succeed. Next to the technical aspects, it is fundamental to determine the acceptance of the robot, which was achieved by analyzing questionnaires given to care workers. Finally, this paper presents lessons learned to help other researchers in similar use-cases.

Object instance detection is a highly relevant task to several robotic applications such as automated order picking, or household and hospital assistance robots. In these applications, a holistic scene labeling is often not required whereas it is sufficient to find a certain object type of interest, e.g. for picking it up. At the same time, large and continuously changing object sets are characteristic in such applications, requiring efficient model update capabilities from the object detector. Today’s monolithic multi-class detectors do not fulfill this criterion for fast and flexible model updates.This paper introduces InstanceNet, an ensemble of efficient single-class instance detectors capable of fast and incremental adaptation to new object sets. Due to a dynamic sampling-based training strategy, accurate detection models for new objects can be obtained within less than 40 minutes on a consumer GPU while only a small percentage of the existing detection models needs to be updated in a very efficient manner. The new detector has been thoroughly evaluated on the basis of a novel dataset of 100 grocery store objects.