Metric depth estimation from visual sensors is crucial for robots to perceive, navigate, and interact with their environment. Traditional range imaging setups, such as stereo or structured light cameras, face hassles including calibration, occlusions, and hardware demands, with accuracy limited by the baseline between cameras. Single- and multi-view monocular depth offers a more compact alternative, but is constrained by the unobservability of the metric scale. Light field imaging provides a promising solution for estimating metric depth by using a unique lens configuration through a single device. However, its application to single-view dense metric depth is under-addressed mainly due to the technology's high cost, the lack of public benchmarks, and proprietary geometrical models and software. Our work explores the potential of focused plenoptic cameras for dense metric depth. We propose a novel pipeline that predicts metric depth from a single plenoptic camera shot by first generating a sparse metric point cloud using a neural network, which is then used to scale and align a dense relative depth map regressed by a foundation depth model, resulting in a dense metric depth. To validate it, we curated the Light Field & Stereo Image Dataset11Dataset available at https://zenodo.org/records/14224205. (LFS) of real-world light field images with stereo depth labels, filling a current gap in existing resources. Experimental results show that our pipeline produces accurate metric depth predictions, laying a solid groundwork for future research in this field.22Work partially supported by the DLR Impulse Project SaiNSOR.

Perceptual aliasing and weak textures pose significant challenges to the task of place recognition, hindering the performance of Simultaneous Localization and Mapping (SLAM) systems. This paper presents a novel model, called UMF (standing for Unifying Local and Global Multimodal Features) that 1) leverages multi-modality by cross-attention blocks between vision and LiDAR features, and 2) includes a re-ranking stage that re-orders based on local feature matching the top-k candidates retrieved using a global representation. Our experiments, particularly on sequences captured on a planetary-analogous environment, show that UMF outperforms significantly previous baselines in those challenging aliased environments. Since our work aims to enhance the reliability of SLAM in all situations, we also explore its performance on the widely used RobotCar dataset, for broader applicability. Code and models are available at https://github.com/DLR-RM/UMF.

Robots with elasticity in structural components can suffer from undesired end-effector positioning imprecision, which exceeds the accuracy requirements for successful manipulation. We present the Probabilistic-Product-Of-Exponentials robot model, a novel approach for kinematic modeling of robots. It does not only consider the robot's deterministic geometry but additionally models time-varying and configuration-dependent errors in a probabilistic way. Our robot model allows to propagate the errors along the kinematic chain and to compute their influence on the end-effector pose. We apply this model in the context of sensor fusion for manipulator pose correction for two different robotic systems. The results of a simulation study, as well as of an experiment, demonstrate that probabilistic configuration-dependent error modeling of the robot kinematics is crucial in improving pose estimation results.

Abstract Region-based methods have become increasingly popular for model-based, monocular 3D tracking of texture-less objects in cluttered scenes. However, while they achieve state-of-the-art results, most methods are computationally expensive, requiring significant resources to run in real-time. In the following, we build on our previous work and develop SRT3D , a sparse region-based approach to 3D object tracking that bridges this gap in efficiency. Our method considers image information sparsely along so-called correspondence lines that model the probability of the object’s contour location. We thereby improve on the current state of the art and introduce smoothed step functions that consider a defined global and local uncertainty. For the resulting probabilistic formulation, a thorough analysis is provided. Finally, we use a pre-rendered sparse viewpoint model to create a joint posterior probability for the object pose. The function is maximized using second-order Newton optimization with Tikhonov regularization. During the pose estimation, we differentiate between global and local optimization, using a novel approximation for the first-order derivative employed in the Newton method. In multiple experiments, we demonstrate that the resulting algorithm improves the current state of the art both in terms of runtime and quality, performing particularly well for noisy and cluttered images encountered in the real world.

In this paper we present DOT (Dynamic Object Tracking), a front-end that added to existing SLAM systems can significantly improve their robustness and accuracy in highly dynamic environments. DOT combines instance segmentation and multi-view geometry to generate masks for dynamic objects in order to allow SLAM systems based on rigid scene models to avoid such image areas in their optimizations.To determine which objects are actually moving, DOT segments first instances of potentially dynamic objects and then, with the estimated camera motion, tracks such objects by minimizing the photometric reprojection error. This short-term tracking improves the accuracy of the segmentation with respect to other approaches. In the end, only actually dynamicmasks are generated.We have evaluated DOT with ORB-SLAM 2 [1] in three public datasets. Our results show that our approach improves significantly the accuracy and robustness of ORB-SLAM2, especially in highly dynamic scenes.

This work aims at accurate estimation of the pose of a close-range 3-D modeling device in real-time, at high-rate, and solely from its own images. In doing so, we replace external positioning systems that constrain the system in size, mobility, accuracy, and cost. At close range, accurate pose tracking from image features is hard because feature projections do not only drift in the face of rotation but also in the face of translation. Large, unknown feature drifts may impede real-time feature tracking and subsequent pose estimation-especially with concurrent operation of other 3-D sensors on the same computer. The problem is solved in Ref. [1] by the partial integration of readings from a backing inertial measurement unit (IMU). In this work we avoid using an IMU by improved feature matching: full utilization of the current state estimation (including structure) during feature matching enables decisive modifications of the matching parameters for more efficient tracking-we hereby follow the Active Matching paradigm.

Accurate online localization is crucial for mobile robotics. In this paper, we describe a real-time image-based localization technique, which is based on a single calibrated camera. This can be supported by a second camera to improve accuracy and to provide the correct translational scale. Our goal is a robust and unbiased pose estimation in highly dynamic scenes on resource-limited systems. The presented approach is characterized through significantly improved robustness of the pose estimation, a novel approach for stereo subpixel accurate landmark initialization, and the speed-up of conventional tracking routines to achieve online capability. Although the algorithm is designed for accurate, online short-range egomotion estimation in hand-held scanning devices, it can be used for any mobile robot application as shown in this paper. Various tests and experimental results with a mobile platform and a hand-held 3D modeler are presented and discussed.

In the context of 3-D scene modeling, this work aims at the accurate estimation of the pose of a close-range 3-D modeling device, in real-time and passively from its own images. This novel development makes it possible to abandon using inconvenient, expensive external positioning systems. The approach comprises an ego-motion algorithm tracking natural, distinctive features, concurrently with customary 3-D modeling of the scene. The use of stereo vision, an inertial measurement unit, and robust cost functions for pose estimation further increases performance. Demonstrations and abundant video material validate the approach.

This paper considers the issue of calibrating a camera with narrow angular field of view using standard, perspective methods in computer vision. In doing so, the significance of perspective distortion both for camera calibration and for pose estimation is revealed. Since narrow angular field of view cameras make it difficult to obtain rich images in terms of perspectivity, the accuracy of the calibration results is expectedly low. From this, we propose an alternative method that compensates for this loss by utilizing the pose readings of a robotic manipulator. It facilitates accurate pose estimation by nonlinear optimization, minimizing reprojection errors and errors in the manipulator transformations at the same time. Accurate pose estimation in turn enables accurate parametrization of a perspective camera.

This paper presents two novel approaches for accurate intrinsic and extrinsic camera calibration. The rationale behind them is the widespread violation of the traditional assumption that the metric structure of the calibration object is perfectly known. A novel formulation parameterizes a checkerboard calibration pattern in such a way that the calibration performs optimally irrespective of its actual dimensions. Simulations and experiments show that it is very rare for traditional calibration methods to come by the accuracy readily attained by this approach.

This paper deals with the concept and implementation of a multi-purpose vision platform. In robotics, numerous applications require perception. A multi-purpose vision platform suited for object recognition, cultural heritage preservation and visual servoing at the same time is missing. In this work, we draw attention to the design principles for such a vision platform. We present its implementation, the 3D-modeller. In specifying and combining multiple sensors, laser-range scanner, laser-stripe profiler and stereo vision, we derive the required mechanical and electrical hardware design. The concepts for synchronization and communication round offs our approach. Precision and frame rate are presented. We illustrate the versatility of the 3D-modeller by addressing four applications: 3D-modeling, exploration, tracking and object recognition. Due to its low weight and generic mechanical interface, it can be mounted on industrial robots, humanoids, or free-handed as well. The 3D-modeller is flexibly applicable, not only in research but also in industry, especially in small batch assembly.

This paper presents a calibration method for eye-in-hand systems in order to estimate the hand-eye and the robot-world transformations. The estimation takes place in terms of a parametrization of a stochastic model. In order to perform optimally, a metric on the group of the rigid transformations SE(3) and the corresponding error model are proposed for nonlinear optimization. This novel metric works well with both common formulations AX=XB and AX=ZB, and makes use of them in accordance with the nature of the problem. The metric also adapts itself to the system precision characteristics. The method is compared in performance to earlier approaches

This paper presents the DLR Laser Stripe Profiler as a component of the DLR multisensory Hand-Guided Device for 3D modeling. After modeling the reconstruction process, we propose a novel method for laser plane self-calibration based on the assessment of the deformations the miscalibration leads to. In addition, the requirement for absence of optical filtering implies the development of a robust stripe segmentation algorithm. Experiments demonstrate the validity and applicability of the approaches.

This article presents a novel gaze control strategy for obstacle avoidance in the context of vision guided humanoid walking. The generic strategy is based on the maximization of the predicted visual information. For information/uncertainty management a new hybrid formulation of an extended Kalman filter is employed. The performance resulting from this view direction control scheme shows the dependence of the intelligent gazing behavior on the pre-planned local path.

This article deals with specific aspects concerning the visual perception process of a humanoid walking machine. An active vision system provides the information about the environment necessary for autonomous goal-oriented locomotion. Due to errors in each stage of the perception process, ideal environment reconstruction is not possible. By modeling these errors, stochastic components can be compensated using a hybrid extended Kalman filter approach with an alternating reference frame, thus reflecting the discontinuous character of biped walking.. The perception results improved by filtering can be used for the autonomous locomotion of the robot. Experiments with the walking machine BARt-UH demonstrate the validity of our approach.

This article deals with the information management for active gaze control in the context of vision-guided humanoid walking. The proposed biologically inspired predictive gaze control strategy is based on the maximization of visual information. The quantification of the information requires a stochastic model of both, the robot and perception system. The information/uncertainty management, i.e. relationships between the system, state estimation and the active measurements, employs a coupled (considering cross-covariances) hybrid (reflecting the discontinuous character of biped walking) extended (copes with nonlinear systems) Kalman filter approach.