The increasing demand in e-commerce, combined with labor shortages and rising wages, is driving the rapid automation of warehouse operations. A critical aspect of this shift is bin packing, where diverse unknown items of varying sizes and shapes must be optimally arranged within a bin or container. Robot bin packing is receiving growing attention and presents unique challenges due to the broad range of objects, packing rules, and task-specific requirements. In response, we propose So-Pack, a generalist packing heuristic for irregularly shaped objects integrated into a flexible, weighted multi-heuristic planning system. The system demonstrates robust performance across general packing scenarios and flexibility to adapt to changing packing rules and specific end-user requirements. Experimental results show that the system outperforms state-of-the-art approaches in key metrics on a new challenging dataset of retail objects in real-world applications.

Robots operating in everyday environments encounter a wide variety of previously unseen objects. Deep Learning methods simplify unknown object and scene segmentation by structuring inherent real-world complexities, improving visual scene understanding. However, they need vast amounts of labeled high-variance data for training. Acquiring these labels for rich real-world data requires significant manual effort, especially for segmentation masks. Although interactive segmentation accelerates this process, these methods still require substantial manual interaction, and the creation of large datasets remains labor-intensive. Consequently, there is a lack of diverse, high-quality datasets for unknown object instance segmentation in everyday environments. This research proposes a semi-automatic, RGB-only algorithmic pipeline for annotating novel objects, reducing manual effort to iteratively placing objects in the scene. We investigate several change detection-based approaches, including remote sensing change detection methods (TTP model), the DeepBackgroundMattingV2 image matting model, and the Segment Anything Model (SAM1 + SAM2) prompted with automatically extracted change regions. We propose the novel ILIS dataset to evaluate these methods in challenging everyday scenes, displaying reliable automatic mask proposal performance of up to 0.9549 mIoU and 0.9565 boundary F1 score. This highlights the potential of this method to accelerate large-scale dataset creation, saving at least 27.27 hours per 1,000 images by eliminating manual annotations.

Efficient object packing is a fundamental challenge in logistics and industrial automation. This work introduces Snuggle-Pack, a novel 3D packing algorithm that integrates Fast Fourier Transform (FFT)-based spatial analysis with a multi-heuristic optimization framework to achieve real-time, high-density packing. Unlike traditional heuristic-based approaches that rely on 2D simplifications, our method operates in a fully 3D volumetric space, ensuring collision-free, stable, and physically feasible placements. At its core, our approach employs a proximity-aware and support-sensitive placement strategy, which encourages objects to fit snugly within their surroundings —hence the name—, optimizing space utilization through ne-grained collision metrics. We evaluate our method on the YCB and IPA-3D1K datasets in both previewed and ad-hoc packing scenarios. Our experiments show that Snuggle-Pack significantly outperforms the state of the art, achieving up to 25% higher packing densities or, alternatively, accelerating computation by up to 10×. Moreover, our framework allows for dynamic adaptation to custom constraints, such as balanced center of mass, weight limitations on fragile items, and safety proximity constraints. These results highlight Snuggle-Pack as an efficient, flexible, and scalable solution for industrial robotic packing tasks.

Robotic applications like automated order picking in warehouses or retail stores, or fetch and carry tasks in hospitals, care homes, or households rely on the capability of service robots to find and handle a specific type of object. These applications are challenging as the set of objects is very large and varies over time. Despite its significance, there is no suitable universal large-scale dataset available from the retail domain, which allows for a principled analysis of all relevant robotics research aspects in that field. Hence, this paper introduces a novel dataset of more than 1,000 retail objects, including color images, 3D scans, and high-resolution textured 3D models of individual objects, synthetic scenes and real settings, which covers the specifics of the retail domain. The dataset was designed to serve researchers in all relevant robotics tasks in retail like 3D reconstruction and object modeling, large-scale object classification and instance detection including incremental learning and fine-grained detection, text reading, logo detection, semantic grounding and affordance detection, grasp analysis and manipulation planning, as well as digital twinning and virtual environments. Based on synthetic RGB images of scenes created from the 3D models, two exemplary use cases are examined in this paper to demonstrate the benefits of the dataset: we evaluate the state-of-the-art incremental object detection method InstanceNet and a few-shot fine-grained object classification method. The results prove the suitability of InstanceNet for incremental object detection on large datasets and are promising for the few-shot object classification system.

For various automated palletizing tasks, the detection of packaging units is a crucial step preceding the actual handling of the packaging units by an industrial robot. We propose an approach to this challenging problem that is fully trained on synthetically generated data and can be robustly applied to arbitrary real world packaging units without further training or setup effort. The proposed approach is able to handle sparse and low quality sensor data, can exploit prior knowledge if available and generalizes well to a wide range of products and application scenarios. To demonstrate the practical use of our approach, we conduct an extensive evaluation on real-world data with a wide range of different retail products. Further, we integrated our approach in a lab demonstrator and a commercial solution will be marketed through an industrial partner.

In this paper, we present a Domain Randomization and a Domain Adaptation approach to transfer experience for entanglement detection and separation from simulation into a real-world bin-picking application. We investigate the influence of different randomization options in image processing and use a CycleGAN as a further Domain Adaptation method to synthesize simulation data as realistically as possible. On the basis of this adapted data we re-train our detection and separation methods and validate the usefulness of these Sim-to-Real methods. In numerous real-world experiments we show that we achieve a significant increase of up to 71.74 % in the performance of the overall system by using the Sim-to-Real approaches as opposed to the direct transfer.

Object instance detection is a highly relevant task to several robotic applications such as automated order picking, or household and hospital assistance robots. In these applications, a holistic scene labeling is often not required whereas it is sufficient to find a certain object type of interest, e.g. for picking it up. At the same time, large and continuously changing object sets are characteristic in such applications, requiring efficient model update capabilities from the object detector. Today’s monolithic multi-class detectors do not fulfill this criterion for fast and flexible model updates.This paper introduces InstanceNet, an ensemble of efficient single-class instance detectors capable of fast and incremental adaptation to new object sets. Due to a dynamic sampling-based training strategy, accurate detection models for new objects can be obtained within less than 40 minutes on a consumer GPU while only a small percentage of the existing detection models needs to be updated in a very efficient manner. The new detector has been thoroughly evaluated on the basis of a novel dataset of 100 grocery store objects.

Single shot approaches have demonstrated tremendous success on various computer vision tasks. Finding good parameterizations for 6D object pose estimation remains an open challenge. In this work, we propose different novel parameterizations for the output of the neural network for single shot 6D object pose estimation. Our learning-based approach achieves state-of-the-art performance on two public benchmark datasets. Furthermore, we demonstrate that the pose estimates can be used for real-world robotic grasping tasks without additional ICP refinement.

This paper introduces a novel approach for the grasping and precise placement of various known rigid objects using multiple grippers within highly cluttered scenes. Using a single depth image of the scene, our method estimates multiple 6D object poses together with an object class, a pose distance for object pose estimation, and a pose distance from a target pose for object placement for each automatically obtained grasp pose with a single forward pass of a neural network.By incorporating model knowledge into the system, our approach has higher success rates for grasping than state-of-the-art model-free approaches. Furthermore, our method chooses grasps that result in significantly more precise object placements than prior model-based work.

Visual dirt detection is becoming an important capability of modern professional cleaning robots both for optimizing their wet cleaning results and for facilitating demand-oriented daily vacuum cleaning. This paper presents a robust, fast, and reliable dirt and office item detection system for these tasks based on an adapted YOLOv3 framework. Its superiority over state-of-the-art dirt detection systems is demonstrated in several experiments. The paper furthermore features a dataset generator for creating any number of realistic training images from a small set of real scene, dirt, and object examples.

In this paper, we introduce a novel learning-based approach for grasping known rigid objects in highly cluttered scenes and precisely placing them based on depth images. Our Placement Quality Network (PQ-Net) estimates the object pose and the quality for each automatically generated grasp pose for multiple objects simultaneously at 92 fps in a single forward pass of a neural network. All grasping and placement trials are executed in a physics simulation and the gained experience is transferred to the real world using domain randomization. We demonstrate that our policy successfully transfers to the real world. PQ-Net outperforms other model-free approaches in terms of grasping success rate and automatically scales to new objects of arbitrary symmetry without any human intervention.

Coverage Path Planning (CPP) describes the process of generating robot trajectories that fully cover an area or volume. Applications are, amongst many others, mobile cleaning robots, lawn mowing robots or harvesting machines in agriculture. Many approaches and facets of this problem have been discussed in literature but despite the availability of several surveys on the topic there is little work on quantitative assessment and comparison of different coverage path planning algorithms. This paper analyzes six popular off-line coverage path planning methods, applicable to previously recorded maps, in the setting of indoor coverage path planning on room-sized units. The implemented algorithms are thoroughly compared on a large dataset of over 550 rooms with and without furniture.

The division of floor plans or navigation maps into single rooms or similarly meaningful semantic units is central to numerous tasks in robotics such as topological mapping, semantic mapping, place categorization, human-robot-interaction, or automatized professional cleaning. Although many map partitioning algorithms have been proposed for various applications there is a lack of comparative studies on these different algorithms. This paper surveys the literature on room segmentation and provides four publicly available implementations of popular methods, which target the semantic mapping domain and are tuned to yield segmentations into complete rooms. In an attempt to provide new users of such technologies guidance in the choice of map segmentation algorithm, those methods are compared qualitatively and quantitatively using several criteria. The evaluation is based on a novel compilation of 20 challenging floor plans.

Millions of office workplaces are cleaned by a surprisingly small group of cleaning workers every day, however, cleaning companies struggle to recruit enough personnel these days. One solution to this challenge is to schedule available professionals for demanding tasks while relieving them from simpler activities which are transferred to a robotic cleaning assistant. Two of such tasks are floor cleaning and waste disposal which account for 70% of the daily cleaning efforts. This paper presents the world's first autonomous cleaning robot prototype that masters both of these tasks and whose development was accompanied by the advice of a large cleaning company. Besides a detailed description of the overall system and its individual components an evaluation is provided based on real world experiments. The results indicate that both cleaning tasks can be solved at high quality but with potential for increased efficiency to meet the required performance. Hence, the paper concludes with a discussion on measures necessary for the development of a commercial prototype.

Accurate surface normal computation is one of the most basic and important tasks for 3d perception. While much progress has been made in speeding up normal estimation algorithms and improving their accuracy, a significant inaccuracy still remains even with modern implementations, which is the correct determination of surface normals close to non-differentiable surface edges. Current algorithms tend to amalgamate neighborhood points from independent surfaces yielding normals that neither fit well to the one nor the other surface. This paper introduces a fast and accurate 3d edge detection algorithm suitable to detect discontinuities both in depth and on surfaces with nearly 90% accuracy at rates beyond 30 Hz. Based on this method, we demonstrate how established normal estimation algorithms can be extended for edge-awareness. Additionally, a new edge-aware, fast, accurate, and robust normal estimation approach is described which exploits the data structures computed for 3d edge detection and estimates normals at 23 Hz. We assess the performance of all proposed methods and compare them with other state-of-the-art approaches.

We present a descriptor estimator for surface-based 3D input data for coarse localization of mobile robots. From the input pointclouds surfaces are reconstructed and simplified to detect stable keypoints which are used to evaluate rotation and translation invariant features. The invariance is achieved by transforming the triangulated input data into the frequency domain by Fourier transformation and spherical harmonics. The pipeline was evaluated against state of the art algorithms and tested to localize a mobile robot. The source code is publicly available.

Derivation of geometric structures from point clouds is an important step towards scene understanding for mobile robots. In this paper, we present a novel method for segmentation and surface classification of ordered point clouds. Data from RGB-D cameras are used as input. Normal based region growing segments the cloud and point feature descriptors classify each segment. Not only planar segments can be described but also curved surfaces. In an evaluation on indoor scenes we show the performance of our approach as well as give a comparison to state of the art methods.

We describe a novel framework that combines an overhead camera and a robot RGB-D sensor for real-time people finding. Finding people is one of the most fundamental tasks in robot home care scenarios and it consists of many components, e.g. people detection, people tracking, face recognition, robot navigation. Researchers have extensively worked on these components, but as isolated tasks. Surprisingly, little attention has been paid on bridging these components as an entire system. In this paper, we integrate the separated modules seamlessly, and evaluate the entire system in a robot-care scenario. The results show largely improved efficiency when the robot system is aided by the localization system of the overhead cameras.

With changes in life expectancy across the world, technologies enhancing well-being of individuals, specifically for older people, are subject to a new stream of research and development. In this paper we present the ACCOMPANY project, a pan-European project which focuses on home companion technologies. The projects aims to progress beyond the state of the art in multiple areas such as empathic and social human-robot interaction, robot learning and memory visualisation, monitoring persons and chores at home, and technological integration of these multiple approaches on an existing robotic platform, Care-O-Bot®3 and in the context of a smart-home environment utilising a multitude of sensor arrays. The resulting prototype from integrating these developments undergoes multiple formative cycles and a summative evaluation cycle towards identifying acceptable behaviours and roles for the robot for example role as a butler or a trainer. Furthermore, the evaluation activities will use an evaluation grid in order to assess achievement of the identified user requirements, formulated in form of distinct scenarios. Finally, the project considers ethical concerns and by highlighting principles such as autonomy, independence, enablement, safety and privacy, it embarks on providing a discussion medium where user views on these principles and the existing tension between some of these principles for example tension between privacy and autonomy over safety, can be captured and considered in design cycles and throughout project developments.

The advances of technologies for mobile robotics enable the application of robots to increasingly complex tasks. Cleaning office buildings on a daily basis is a problem that could be partially automatized with a cleaning robot that assists the cleaning professional yielding a higher cleaning capacity. A typical task in this domain is the selective cleaning, that is a focused cleaning effort to dirty spots, which speeds up the overall cleaning procedure significantly. To enable a robotic cleaner to accomplish this task, it is first necessary to distinguish dirty areas from the clean remainder. This paper discusses a vision-based dirt detection system for mobile cleaning robots that can be applied to any surface and dirt without previous training, that is fast enough to be executed on a mobile robot and which achieves high dirt recognition rates of 90% at an acceptable false positive rate of 45%. The paper also introduces a large database of real scenes which was used for the evaluation and is publicly available.

At the core of every object recognition system lies the development and integration of distinct feature descriptors to create object representations robust against varying perspectives or lightning conditions. Recent work has primarily focused on the development of distinct point features. While these features achieve impressive recognition results, point features fail to capture the shape and appearance of an object with less or even without texture. This paper proposes a novel method for the rapid and dense computation of 2D and 3D image cues from RGB-D data to target the recognition of objects without rich texture and a global histogram-based descriptor for the distinct description of object models.

In this paper, we present a novel method for surface reconstruction with a low execution time for segmenting and representing scattered scenes accurately. The surfaces are described in a memory-efficient fashion as polynomial functions and polygons. Segmentation and parameter determination is done in one pass by using a quadtree on ordered point clouds, which results in a complexity of O(log n). This paper includes an evaluation with respect to reconstruction accuracy, segmentation precision, execution time and compression ratio of everyday indoor scenes. Our surface reconstruction algorithm outperforms comparable approaches with respect to execution time and accuracy. More importantly, the new technique handles curved shapes accurately and enables complex tasks like 3D mapping for mobile robots in an unknown environment.

This paper investigates existing methods for 3D point feature description with a special emphasis on their expressiveness of the local surface geometry. We choose three promising descriptors, namely Radius-Based Surface Descriptor (RSD), Principal Curvatures (PC) and Fast Point Feature Histograms (FPFH), and present an approach for each of them to show how they can be used to classify primitive local surfaces such as cylinders, edges or corners in point clouds. Furthermore these descriptor-classifier combinations have to hold an in-depth evaluation to show their discriminative power and robustness in real world scenarios. Our analysis incorporates detailed accuracy measurements on sparse and noisy point clouds representing typical indoor setups for mobile robot tasks and considers the resource consumption to assure real-time processing.

We present a novel algorithm that solves the distributor's pallet packing problem. In contrast to existing algorithms, our method optimizes stack stability in addition to stack volume. Furthermore, our algorithm explicitly handles cases where the construction of homogeneous layers of packages with equal height is impossible due to differences in package heights and quantities. The algorithm is a nested beam search that separately optimizes local and global evaluation criteria. We show successful results on both real world and synthetic data sets, compare our performance to an existing algorithm and demonstrate experimental applications in simulation and on a real palletizing robot.