Tracking the pose of a camera is at the core of visual localization methods used in many applications. As the observations of a camera are inherently affected by lighting, it has always been a challenge for these methods to cope with varying lighting conditions. Thus far, this issue has mainly been approached with the intent to increase robustness by choosing lighting invariant map representations. In contrast, our work aims at explicitly exploiting lighting effects for camera tracking. To achieve this, we propose a lighting adaptable map representation for indoor environments that allows real-time rendering of the scene illuminated by an arbitrary subset of the lamps contained in the model. Our method for estimating the light setting from the current camera observation enables us to adapt the model according to the lighting conditions present in the scene. As a result, lighting effects like cast shadows do no longer act as disturbances that demand robustness but rather as beneficial features when matching observations against the map. We leverage these capabilities in a direct dense camera tracking approach and demonstrate its performance in realworld experiments in scenes with varying lighting conditions.

Indoor localization is one of the crucial enablers for deployment of service robots. Although several successful techniques for indoor localization have been proposed, the majority of them relies on maps generated from data gathered with the same sensor modality used for localization. Typically, tedious labor by experts is needed to acquire this data, thus limiting the readiness of the system as well as its ease of installation for inexperienced operators. In this paper, we propose a memory and computationally efficient monocular camera-based localization system that allows a robot to estimate its pose given an architectural floor plan. Our method employs a convolutional neural network to predict room layout edges from a single camera image and estimates the robot pose using a particle filter that matches the extracted edges to the given floor plan. We evaluate our localization system using multiple real-world experiments and demonstrate that it has the robustness and accuracy required for reliable indoor navigation.

Robust data association is necessary for virtually every SLAM system and finding corresponding points is typically a preprocessing step for scan alignment algorithms. Traditionally, handcrafted feature descriptors were used for these problems but recently learned descriptors have been shown to perform more robustly. In this work, we propose a local feature descriptor for 3D LiDAR scans. The descriptor is learned using a Convolutional Neural Network (CNN). Our proposed architecture consists of a Siamese network for learning a feature descriptor and a metric learning network for matching the descriptors. We also present a method for estimating local surface patches and obtaining ground-truth correspondences. In extensive experiments, we compare our learned feature descriptor with existing 3D local descriptors and report highly competitive results for multiple experiments in terms of matching accuracy and computation time.

The ability to build models of the environment is an essential prerequisite for many robotic applications. In recent years, mapping of dense surface geometry using RGB-D cameras has seen extensive progress. Many approaches build colored models, typically directly using the intensity values provided by the camera. Unfortunately, these intensities are inherently affected by illumination. Therefore, the resulting maps only represent the environment for one specific lighting condition. To overcome this limitation, we propose to build reflectance maps that are invariant against changes in lighting. Our approach estimates the diffuse reflectance of a surface by recovering its radiosity and the corresponding irradiance. As imperfections in this process can significantly degrade the reflectance estimate, we remove outliers in the high dynamic range radiosity estimation and propose a method to refine the reflectance estimate. Our system implements the whole pipeline for offline reconstruction of dense reflectance maps including the segmentation of light emitters in the scene. We demonstrate the applicability of our approach in real-world experiments under varying lighting conditions.

Modern automation demands mobile robots to be robustly localized in complex scenarios. Current localization systems typically use maps that require to be built and interpreted by experienced operators, increasing deployment costs as well as reducing the adaptability of robots to rearrangements in the environment. In contrast, architectural floor plans can be easily understood by non-expert users and typically represent only the non-rearrangeable parts of buildings. In this paper we propose a system for robot localization in architectural CAD drawings. Our method employs a simultaneous localization and mapping approach to online augment the floor plan with a map represented as a pose-graph with LiDAR measurements. Whenever the environment is accurately mapped in the vicinity of the robot, we use the graph to perform relative localization. We thoroughly evaluate our system in challenging real-world scenarios. Experiments demonstrate that our method is able to robustly track the robot pose even when the floor plan shows major discrepancies from the real-world. We show that our system achieves sub-centimeter accuracy and is suitable for real-time application.

Robots are expected to operate autonomously in increasingly complex scenarios such as crowded streets or heavy traffic situations. Perceiving the dynamics of moving objects in the environment is crucial for safe and smart navigation and therefore a key enabler for autonomous driving. In this paper we present a novel model-free approach for detecting and tracking dynamic objects in 3D LiDAR scans obtained by a moving sensor. Our method only relies on motion cues and does not require any prior information about the objects. We sequentially detect multiple motions in the scene and segment objects using a Bayesian approach. For robustly tracking objects, we utilize their estimated motion models. We present extensive quantitative results based on publicly available datasets and show that our approach outperforms the state of the art.

Localizing a camera in a given map is essential for vision-based navigation. In contrast to common methods for visual localization that use maps acquired with cameras, we propose a novel approach, which tracks the pose of monocular camera with respect to a given 3D LiDAR map. We employ a visual odometry system based on local bundle adjustment to reconstruct a sparse set of 3D points from image features. These points are continuously matched against the map to track the camera pose in an online fashion. Our approach to visual localization has several advantages. Since it only relies on matching geometry, it is robust to changes in the photometric appearance of the environment. Utilizing panoramic LiDAR maps additionally provides viewpoint invariance. Yet low-cost and lightweight camera sensors are used for tracking. We present real-world experiments demonstrating that our method accurately estimates the 6-DoF camera pose over long trajectories and under varying conditions.

The perception of the dynamic aspects of the environment is a highly relevant precondition for the realization of autonomous robot system acting in the real world. In this paper, we propose a novel method for estimating dense rigid scene flow in 3D LiDAR scans. We formulate the problem as an energy minimization problem, where we assume local geometric constancy and incorporate regularization for smooth motion fields. Analyzing the dynamics at point level helps in inferring the fine-grained details of motion. We show results on multiple sequences of the KITTI odometry dataset, where we seamlessly estimate multiple motions pertaining to different dynamic objects. Furthermore, we test our approach on a dataset with pedestrians to show how our method adapts to a case with non-rigid motion. For comparison we use the ground truth from KITTI and show how our method outperforms different ICP-based methods.