Global point cloud registration is an essential module for localization, of which the main difficulty exists in estimating the rotation globally without initial value. With the aid of gravity alignment, the degree of freedom in point cloud registration could be reduced to 4DoF, in which only the heading angle is required for rotation estimation. In this paper, we propose a fast and accurate global heading angle estimation method for gravity-aligned point clouds. Our key idea is that we generate a translation invariant representation based on Radon Transform, allowing us to solve the decoupled heading angle globally with circular cross-correlation. Besides, for heading angle estimation between point clouds with different distributions, we implement this heading angle estimator as a differentiable module to train a feature extraction network end-to-end. The experimental results validate the effectiveness of the proposed method in heading angle estimation and show better performance compared with other methods.

Accurate LiDAR-camera extrinsic calibration is a precondition for many multi-sensor systems in mobile robots. Most calibration methods rely on laborious manual operations and calibration targets. While working online, the calibration methods should be able to extract information from the environment to construct the cross-modal data association. Convolutional neural networks (CNNs) have powerful feature extraction ability and have been used for calibration. However, most of the past methods solve the extrinsic as a regression task, without considering the geometric constraints involved. In this paper, we propose a novel end-to-end extrinsic calibration method named DXQ-Net, using a differentiable pose estimation module for generalization. We formulate a probabilistic model for LiDAR-camera calibration flow, yielding a prediction of uncertainty to measure the quality of LiDAR-camera data association. Testing experiments illustrate that our method achieves a competitive with other methods for the translation component and state-of-the-art performance for the rotation component. Generalization experiments illustrate that the generalization performance of our method is significantly better than other deep learning-based methods.

Localization is fundamental for autonomous vehicle applications. Compared with widely developed LiDAR-based vehicle localization, vision based localization has attracted considerable attention in recent years owing to the low-cost sensor. One of the challenges for visual localization is the outlier in measurements due to the appearance changes caused by illumination, season, and weather. To address this problem, we present a real-time robust visual localization approach that achieves deterministic optimality with global convergence. The idea is to decouple the rotation and translation estimation by utilizing the fact that the pitch and roll angles of the query pose are similar to those of the map reference pose, since the vehicle motion is locally planar. Based on the decoupled formulation, we first estimate the optimal yaw angle and eliminate the majority of outliers by an efficient inlier voting method, then find the optimal translation by maximum clique search. The two subproblem estimators are embedded into a prioritized search paradigm to guarantee deterministic optimality. In the experiments, the simulation demonstrates that the proposed method can achieve superior robustness even dealing with extreme outlier rates (95%). Results on both public and self-collected real-world vehicle datasets validate the effectiveness of the proposed method in the real application.

Visual localization for planar moving robot is important to various indoor service robotic applications. To handle the textureless areas and frequent human activities in indoor environments, a novel robust visual localization algorithm which leverages dense correspondence and sparse depth for planar moving robot is proposed. The key component is a minimal solution which computes the absolute camera pose with one 3D-2D correspondence and one 2D-2D correspondence. The advantages are obvious in two aspects. First, the robustness is enhanced as the sample set for pose estimation is maximal by utilizing all correspondences with or without depth. Second, no extra effort for dense map construction is required to exploit dense correspondences for handling textureless and repetitive texture scenes. That is meaningful as building a dense map is computational expensive especially in large scale. Moreover, a probabilistic analysis among different solutions is presented and an automatic solution selection mechanism is designed to maximize the success rate by selecting appropriate solutions in different environmental characteristics. Finally, a complete visual localization pipeline considering situations from the perspective of correspondence and depth density is summarized and validated on both simulation and public real-world indoor localization dataset.

Large scale 3D maps constructed via LiDAR sensor are widely used on intelligent vehicles for localization in outdoor scenes. However, loading, communication and processing of the original dense maps are time consuming for onboard computing platform, which calls for a more concise representation of maps to reduce the complexity but keep the performance of localization. In this paper, we propose a teacher-student learning paradigm to compress the 3D point cloud map. Specifically, we first find a subset of LiDAR points with high number of observations to preserve the localization performance, which is regarded as the teacher of map compression. An efficient optimization strategy is proposed to deal with the massive data in original map. With the supervision of compressed map, a student model is built by training a random forest model fed with geometric feature descriptors of each point. As a result, the student model is able to compress the map without referring to the expensive numerical optimization. Additionally, by incorporating the features, the innovative student model can be generalized to other new maps while no re-training is required. We conduct thorough experiments on multi-session dataset and KITTI dataset to demonstrate the effectiveness and efficiency of the proposed learning paradigm, and the comparison with other map compression methods. The final results show that the learned student model can achieve efficient map compression with comparable LiDAR based localization performance to the original map at the same time.

When robots move autonomously for long-term, varied appearance such as the transition from day to night and seasonal variation brings challenges to visual place recognition. Defining an appearance condition (e.g. a season, a kind of weather) as a domain, we consider that the desired representation for place recognition (i) should be domain-unrelated so that images from different time can be matched regardless of varied appearance, (ii) should be learned in a self-supervised manner without the need of massive manually labeled data, and (iii) should be able to train among multiple domains in one model to keep limited model complexity. This paper sets to find domain-unrelated features across extremely changing appearance, which can be used as image descriptors to match between images collected at different conditions. We propose to use the adversarial network to disentangle domain-unrelated and domain-related features, which are named place and appearance features respectively. During training, only domain information is needed without requiring manually aligned image sequences. Experiments demonstrated that our method can disentangle place and appearance features in both toy case and images from the real world, and the place feature is qualified in place recognition tasks under different appearance conditions. The proposed network is also adaptable to multiple domains without increasing model capacity and shows favorable generalization.

Autonomous mobile vehicles are expected to perform persistent and accurate localization with low-cost equipment. To achieve this goal, we propose a stereo camera based visual localization method using a modified laser map, which takes the advantage of both the low cost of camera, and high geometric precision of laser data to achieve long-term performance. Considering that LiDAR and camera give measurements of the same environment in different modalities, the cross-modal invariance is investigated to modify the laser map for visual localization. Specifically, a map learning algorithm is introduced to sample the robust subsets in laser maps that are useful for visual localization using multi-session visual and laser data. Further, a generative map model is derived to describe this cross-modal invariance, based on which two types of measurements are defined to model the laser map points as appropriate visual observations. Tightly coupling these measurements within the local bundle adjustment during online sliding-window based visual odometry, the vehicle can achieve robust localization even one year after the map was built. The effectiveness of the proposed method is evaluated on both the public KITTI datasets and self-collected datasets in our campus, which include seasonal, illumination and object variations. On all experimental localization sessions, our method provides satisfactory results, even when the direction is opposite to that in the mapping session, verifying the superior performance of the laser map based visual localization method.

Global localization in 3D point clouds is a challenging task for mobile vehicles in outdoor scenarios, which requires the vehicle to localize itself correctly in a given map without prior knowledge of its pose. This is a critical component of autonomous vehicles or robots on the road for handling localization failures. In this paper, based on reduced dimension scan representations learned from neural networks, a solution to global localization is proposed by achieving place recognition first and then metric pose estimation in the global prior map. Specifically, we present a semi-handcrafted feature learning method for 3D Light detection and ranging (LiDAR) point clouds using artificial statistics and siamese network, which transforms the place recognition problem into a similarity modeling problem. Additionally, the sensor data using dimension reduced representations require less storage space and make the searching easier. With the learned representations by networks and the global poses, a prior map is built and used in the localization framework. In the localization step, position only observations obtained by place recognition are used in a particle filter algorithm to achieve precise pose estimation. To demonstrate the effectiveness of our place recognition and localization approach, KITTI benchmark and our multi-session datasets are employed for comparison with other geometric-based algorithms. The results show that our system can achieve both high accuracy and efficiency for long-term autonomy.

Visual localization is one of the primary capabilities for mobile robots. Long-term visual localization in real time is particularly challenging, in which the robot is required to efficiently localize itself using visual data where appearance may change significantly over time. In this paper, we propose a cloud-based visual localization system targeting at long-term localization in real time. On the robot, we employ two estimators to achieve accurate and real-time performance. One is a sliding-window based visual inertial odometry, which integrates constraints from consecutive observations and self-motion measurements, as well as the constraints induced by localization results from the cloud. This estimator builds a local visual submap as the virtual observation which is then sent to the cloud as new localization constraints. The other one is a delayed state Extended Kalman Filter to fuse the pose of the robot localized from the cloud, the local odometry and the high-frequency inertial measurements. On the cloud, we propose a longer sliding-window based localization method to aggregate the virtual observations for larger field of view, leading to more robust alignment between virtual observations and the map. Under this architecture, the robot can achieve drift-free and real-time localization using onboard resources even in a network with limited bandwidth, high latency and existence of package loss, which enables the autonomous navigation in real-world environment. We evaluate the effectiveness of our system on a dataset with challenging seasonal and illuminative variations. We further validate the robustness of the system under challenging network conditions.

Visual localization has attracted considerable attention due to its low-cost and stable sensor, which is desired in many applications, such as autonomous driving, inspection robots and unmanned aerial vehicles. However, current visual localization methods still struggle with environmental changes across weathers and seasons, as there is significant appearance variation between the map and the query image. The crucial challenge in this situation is that the percentage of outliers, i.e. incorrect feature matches, is high. In this paper, we derive minimal closed form solutions for 3D-2D localization with the aid of inertial measurements, using only 2 point matches or 1 point match and 1 line match. These solutions are further utilized in the proposed 2-entity RANSAC, which is more robust to outliers as both line and point features can be used simultaneously and the number of matches required for pose calculation is reduced. Furthermore, we introduce three feature sampling strategies with different advantages, enabling an automatic selection mechanism. With the mechanism, our 2-entity RANSAC can be adaptive to the environments with different distribution of feature types in different segments. Finally, we evaluate the method on both synthetic and real-world datasets, validating its performance and effectiveness in inter-session scenarios.

Long-term visual localization in outdoor environment is a challenging problem, especially faced with the cross-seasonal, bi-directional tasks and changing environment. In this paper we propose a novel visual inertial localization framework that localizes against the LiDAR-built map. Based on the geometry information of the laser map, a hybrid bundle adjustment framework is proposed, which estimates the poses of the cameras with respect to the prior laser map as well as optimizes the state variables of the online visual inertial odometry system simultaneously. For more accurate crossmodal data association, the laser map is optimized using multisession laser and visual data to extract the salient and stable subset for visual localization. To validate the efficiency of the proposed method, we collect data in south part of our campus in different seasons, along the same and opposite-direction route. In all sessions of localization data, our proposed method gives satisfactory results, and shows the superiority of the hybrid bundle adjustment and map optimization1.

Global localization in 3D point clouds is a challenging problem of estimating the pose of vehicles without any prior knowledge. In this paper, a solution to this problem is presented by achieving place recognition and metric pose estimation in the global prior map. Specifically, we present a semi-handcrafted representation learning method for LiDAR point clouds using siamese LocNets, which states the place recognition problem to a similarity modeling problem. With the final learned representations by LocNet, a global localization framework with range-only observations is proposed. To demonstrate the performance and effectiveness of our global localization system, KITTI dataset is employed for comparison with other algorithms, and also on our long-time multi-session datasets for evaluation. The result shows that our system can achieve high accuracy.