We present a systematic approach called Extended VINS-Mono to utilize VINS-Mono, a state-of-the-art monocular visual-inertial relative localization method, targeting practical vehicle localization in large-scale outdoor road environments. Our proposed fusion approach associates multiple independent localization methods and provides multiple (projected) state estimates in a desired coordinate system to satisfy different accuracy, rate and latency requirements. We extend VINS-Mono with absolute localization methods like GNSS and relative localization methods like Kalman-filter-based INS to provide global state estimation for navigation/routing and local state estimation for planning/control. Additionally, Extended VINS-Mono addresses two significant drawbacks in VINS-Mono for use in large-scale outdoor road environments. First, motion on an almost planar road surface will make scale unobservable in VINS-Mono. Secondly, moving objects in dynamic scenarios will degrade accuracy. We handle the scale estimation problem of VINS-Mono by extending its (re-)initialization process with speed readings and introducing a speed factor for use with graph optimization. A dynamic feature-point filter method with masks from DNN-based object detection handles dynamic environments and re-collects feature points on stationary objects like parked cars. Better global accuracy is obtained with Extended VINS-Mono, compared to VINS-Mono, in a 25 km-trip journey through highways, tunnels, urban areas and suburban areas in Pittsburgh. Thus, Extended VINS-Mono can be used for reliable and accurate absolute localization in dynamic road environments. We also evaluate the accuracy, localization rate and latency of multiple (projected) state estimates in the global coordinate system from multiple localization methods. Our fusion method is therefore able to satisfy different localization requirements of various tasks on an intelligent vehicle.

A monocular VIO (Visual-Inertial Odometry) system provides a compact, low-cost, and easily-deployed configuration for relative localization. However, using thermal vision for VIO is much less studied than using a visible-spectrum camera. A thermal-vision camera works under all lighting conditions and has been used to detect pedestrians, cars and animals at nighttime to provide ADAS (Advanced Driver Assistance System) functions. Common problems in directly using thermal images in conventional VIO methods are: (1) lower signal-to-noise ratio and fewer reliable feature points in the texture-less thermal images, (2) periodic recalibration hampers thermal image capturing and feature-point tracking, and (3) the “jello” effect of the rolling shutter readout architecture is sensitive to aggressive vehicle maneuvers. Extended VINS-Mono, proposed in our previous work [1], aims at providing relative and absolute localization of a vehicle in large-scale outdoor road environments by introducing (1) absolute localization methods to enable VINS-Mono to output local and global state estimates simultaneously, (2) vehicle speed readings for fast (re-)initialization and reliable scale estimates, and (3) DNN-based object detection methods to remove nonstationary objects from the visible scene. In this paper, we show that Extended VINS-Mono can use thermal images to provide relative and absolute localization even when light conditions are very poor. We conducted several experiments on a 25 Km-trip journey through highways, tunnels, urban areas and suburban areas in Pittsburgh, USA during daytime and nighttime to evaluate the performance of Extended Thermal VINS-Mono, including (re-)initialization, accuracy, rate, and latency. Our evaluation confirms that using thermal vision for localization satisfies localization requirements in large-scale outdoor road environments when the visible-spectrum camera performs poorly.

A precise and efficient tracking technique is essential for an intelligent vehicle to interact with the surrounding vehicles on the road. In this paper, we propose a model-based vehicle tracking method using the Rao-Blackwellized particle filter (RBPF) and scaling series particle filter (SSPF).

Obstacle detection is an essential technique for intelligent vehicles. Environmental sensing especially plays a vital role to achieve accurate obstacle detection. Unlike classical 2D scan, emerging 3D Light Detection and Ranging (LiDAR) sensors can scan dense point cloud at one time, which represents detailed information of urban environments. The downside of obstacle detection using 3D LiDAR, on the other hand, is its computational cost posed by a large amount of 3D data. The virtual scan (VScan), first introduced by Petrovskaya et al. [1] for efficient vehicle detection and tracking, is a 2D compression of 3D point cloud to represent free space, obstacles and unknown areas. To overcome the computational problem of obstacle detection using 3D LiDAR, therefore, VScan is suitable. In addition, it can bridge across new-born 3D LiDAR sensors and many matured applications based on 2D scan, including occupancy grid map, SLAM, planning, detection, and tracking, due to its 2D representation of 3D point cloud.

Many robotic and mobile mapping systems have been developed using multiple 2D LIDARs (briefly multi-LIDAR system) to sense environment. In such systems, extrinsic calibration of all LIDARs is essential for making collaborative use of the data from different sensors. This research aims at developing a calibration method for multi-LIDAR systems at the general scene, such as an outdoor place or an underground parking-lot, without modification to environment by putting calibration targets. In this paper, the calibration method is proposed by aligning the 3D data of different LIDARs. They are concerned at two-levels: 1) reference calibration, i.e. finding the transformation from a reference LIDAR to the platform frame; 2) multi-LIDAR calibration, i.e. finding the LIDARs' relative geometries by referring to the reference one. The method is examined in calibrating the multiple 2D LIDARs on an intelligent vehicle platform POSS-V, where the data collected through a driving in an underground parking-lot are registered to find sensors' geometry. Calibration accuracy is examined by comparing with a CAD model of the scene, which was measured by using a total station.

It has become a well-known technology that 3D measurement of a large environment could be achieved by using a number of 2D LIDARs on a mobile platform. In such a system, calibration is essential for making collaborative use of different LIDAR data, while existing methods usually require modifications to the environments, such as putting calibration targets, or rely on special facilities, which is labor intensive and put many restrictions to potential applications. This research aims at developing a calibration method for multiple 2D LIDAR sensing systems, which could be conducted in a general outdoor environment using the features of a nature scene. Special focus is cast on solving the noisy sensing in a complex environment and the occlusions caused by largely different sensor viewpoints. A multi-type geometric feature based calibration algorithm is proposed, which extracts the features such as points, lines, planes and quadrics from the 3D points of each LIDAR sensing. Transformation parameters from each sensor to the frame on a moving platform is estimated by matching the multi-type features. Experiments are conducted using the data sets of an intelligent vehicle platform (POSS-V) through a driving in the campus of Peking University. Results of calibrating two LIDAR sensors with largely different viewpoints are presented, and the accuracy and robustness concerning noisy feature extractions are examined intensively.

It becomes a well-known technology that a low-level map of complex environment containing 3D laser points can be generated using a robot with laser scanners. Given a cloud of 3D laser points of an urban scene, this paper proposes a method for locating the objects of interest, e.g. traffic signs or road lamps, by computing object-based saliency. Our major contributions are: 1) a method for extracting simple geometric features from laser data is developed, where both range images and 3D laser points are analyzed; 2) an object is modeled as a graph used to describe the composition of geometric features; 3) a graph matching based method is developed to locate the objects of interest on laser data. Experimental results on real laser data depicting urban scenes are presented; efficiency as well as limitations of the method are discussed.