Simultaneous localization and mapping (SLAM) and moving object detection and tracking (MODT) are two fundamental problems for autonomous driving systems. Multi-vehicle cooperative SLAM and cooperative object perception, which take advantage of multi-vehicle information sharing, can overcome inherent limitations of single vehicle perception, such as view occlusion. Solutions to SLAM and MODT usually rely on certain assumptions, such as the static environment assumption for SLAM and the accurate ego-vehicle pose assumption for MODT. However, it is difficult or even impossible to have these assumptions hold in complex dynamic environments. We propose a LiDAR-based coupled cooperative simultaneous SLAM and MODT (C-SLAMMODT) strategy, which not only handles the SLAM and tracking problems in dynamic environments but also overcomes limitations of single vehicle perception. The proposed C-SLAMMODT outperforms both cooperative SLAM and cooperative MODT. This method includes a cooperative SLAM module that can augment ego-vehicle pose estimation by shared information from neighbouring vehicles, a cooperative MODT module that applies a state-of-the-art adaptive feature-level fusion model to fuse multi-vehicle data, improving detection precision and overcoming the limitations of perception in occlusion situations. Furthermore, a unified factor graph optimization integrates the information obtained from ego-vehicle states, neighbor-vehicle shared data, and dynamic object states to augment pose estimation and realize object tracking. Various comparative experiments demonstrate the performance and advantages of the proposed C-SLAMMODT solution in terms of accuracy and robustness.

Attributed to the unstructured and sparse nature of point clouds, the transformer shows greater potential in point clouds data processing. However, the recent query-based 3D detectors usually project the features acquired from a sparse backbone into the structured and compact Bird’s Eye View(BEV) plane before adopting the transformer, which destroys the sparsity of features, introducing empty tokens and additional resource consumption for the transformer. To this end, in this paper, we propose a novel query-based 3D detector called Clusterformer, our Clusterformer regards each object as a cluster of 3D space which mainly consists of the non-empty voxels belonging to the same object, and leverages the cluster to conduct the transformer decoder to generate the proposals from the sparse voxel features directly. Such cluster-based transformer structure can effectively improve the performance and convergence speed of query-based detectors by making use of the object prior information contained in the clusters. Additionally, we introduce a Query2Key strategy to enhance the key and value features with the object-level information iteratively in our cluster-based transformer structure. Experimental results show that the proposed Clusterformer outperforms the previous query-based detectors with a lower latency and memory usage, which achieves state-of-the-art performance on the Waymo Open Datasets and KITTI Datasets.

In autonomous driving, localization is very important to ensure the accuracy of all the other tasks, and multi- vehicle cooperative localization is a popular research direction in recent years for its improvement on state estimation. However, due to the complex environment, complete information about relative pose for cooperative localization could sometimes be hard to be obtained. Since the inter-vehicle distance is easily measured, the use of inter-vehicle distance for multi-vehicle cooperative localization has become a meaningful topic. In this paper, we propose a general decentralized framework for the inter-vehicle distance aided cooperative localization. In order to fuse the inter-vehicle distance with the state estimation of ego-vehicle, the split covariance intersection filter (Split CIF) is used as the data fusion method. Then, we verify the advantages of the proposed method through simulation-based experiments, demonstrate its applicable conditions and provide its practical application scenarios.

High-precision localization in an unknown environment is the fundamental requirement of autonomous vehicle for safe advanced driving. Cooperative localization has obvious advantages in precision, fault tolerance, and flexibility compared with single-vehicle localization, which estimates the vehicle pose via fusing the multiple sources data from sensors and extra shared neighbor vehicle information available when multiple vehicles operate simultaneously. For cooperative localization, there are more than one type of error data sources that degenerate the state estimation need to be considered, such as inter-estimation correlation and innovation, observation outliers, which may exist simultaneously. Previous works usually evaluate performance under one of the error types at one time, or they do not perform well under some extreme situations where both of the above-mentioned correlation and outliers exist. This paper presents an accurate and robust iterated split covariance intersection filter (Iterated Split CIF) based cooperative localization strategy with a decentralized framework, which can ensure the performance when data sources of various error types exist at the same time. In addition, we adopt an effective point cloud registration method to obtain the cooperative relative pose estimation using mutually shared information from neighbor vehicles. A CARLA simulator based comparative study demonstrates the potential and advantage of the proposed multi-vehicle cooperative localization using Iterated Split CIF in terms of accuracy, robustness and efficiency.

A deraining network can be interpreted as a conditional generator that aims at removing rain streaks from image. Most existing image deraining methods ignore model errors caused by uncertainty that reduces embedding quality. Unlike existing image deraining methods that embed low-quality features into the model directly, we replace low-quality features by latent high-quality features. The spirit of closed-loop feedback in the automatic control field is borrowed to obtain latent high-quality features. A new method for error detection and feature compensation is proposed to address model errors. Extensive experiments on bench-mark datasets as well as specific real datasets demonstrate that the proposed method outperforms recent state-of-the-art methods. Code is available at: https://github.com/LI-Hao-SJTU/DerainRLNet

Initial global localization is important to mobile robotics in terms of navigation initialization (or re-initialization) and loop closure in SLAM. 3D LiDARs are commonly used for mobile robotics, yet LiDAR-based initial global localization (especially at large scale such as in outdoor environments) is still challenging due to lack of salient features in LiDAR range data. Inspired by visual SLAM oriented initial global localization methods, we propose a method of LiDAR-based initial global localization using 2D submap projection image (SPI). For this, global descriptors from SPIs are extracted for place recognition; pose estimation is realized by feature point matching between the queried SPI and SPIs from a global map database. The proposed initial global localization module runs at 2.4 Hz with precision of 1.2 m and for translation and 1.2° for rotation, which can serve as a suitable initial estimate for subsequent pose estimation refinement via existing mature point cloud registration methods.

Simultaneous localization and mapping (SLAM) is important to outdoor intelligent vehicle applications. Multivehicle cooperative SLAM which takes advantage of data sharing can outperform single vehicle SLAM. This paper proposes a multi-vehicle cooperative SLAM method using iterated split covariance intersection filter (Iterated Split CIF). In the proposed method, a vehicle can flexibly perform cooperative SLAM with other vehicles in decentralized way, without complicated monitoring and controlling of data flow. Moreover, the innovation and observation outliers caused by modeling errors or abnormal measurements can be solved reliably. A simulation-based comparative study demonstrates the potential and advantage of the proposed multi-vehicle cooperative SLAM using Iterated Split CIF in terms of accuracy and robustness.

Since fish-eye cameras are popular in the intelligent transportation systems, accurate pedestrian detection in fish-eye images becomes more and more critical in low-speed scenarios. Usually, big data based training is the key for detectors to handle the distortion problem in fish-eye images. Especially, hard examples are more important for detectors. They have more complex features and are hard to recognize. In conventional methods, fish-eye images are collected and labeled manually. These methods are expensive and labor-intense. More importantly, these methods are still hard to collect abundant hard examples since they are rare in reality. This work proposes the Distortion Generation Network, which generates generous fish-eye images automatically using only small samples. Moreover, the Adversarial Distortion Generation Network is proposed to mine hard examples via adversarial training. These hard examples benefit detectors to be more robust to seriously distorted objects in fish-eye images. Experiments with the ETH, the KITTI, the GM-ATCI and real fish-eye datasets demonstrate that the proposed methods achieve higher accuracy than conventional methods in pedestrian detection in fish-eye images.

Imbalanced training is a challenge in the field of autonomous driving. For the steering estimation task, imbalanced training is the core reason that an end-to-end model cannot estimate sharp steering value well. Inspired by researches on the steering estimation, this paper proposes a novel loss function to train a high-performance end-to-end model for handling the imbalanced training problem, which is named SteeringLoss. Firstly, the imbalanced distribution of the steering value for driving datasets is analyzed, which is similar to the double long-tailed distribution. Secondly, with the feature of distribution, this paper designs a cost-sensitive loss function step by step, the new loss function can improve the impact of the sharp steering value while maintaining the impact of the small steering value. Thirdly, three typical steering estimation models are established with SteeringLoss for demonstration, including the CNN model, the CNN-LSTM model and the 3DCNN-LSTM model. Finally, three experiments are designed for SteeringLoss: Experiment I demonstrates SteeringLoss can avoid imbalanced training problem; Experiment II discusses the finetuning principle of SteeringLoss and gives the basic guideline for using SteeringLoss; Experiment III shows the results on different typical end-to-end steering estimation models, which shows effectiveness of SteeringLoss for all the models and gives different solutions for the steering estimation. Moreover, the SteeringLoss is suitable for the imbalanced training with similar distributions of datasets besides end-to-end steering estimation, which indicates the potential value of the research on SteeringLoss in the future.

Inter-vehicle relative localization is essential to various multi-vehicle collaborative tasks such as environment perception augmentation, long range path planning and communication-assisted driving applications. We propose a real-time collaborative relative localization framework to estimate inter-vehicle poses via vision and LiDAR sensors. The vision sensor is used for inter-vehicle pose initialization, whereas the LiDAR sensor is used for SLAM and pose refinement. A comparative study between our proposed method and a representative baseline method based on ORB-SLAM2-stereo is performed in the CARLA simulator, which demonstrates advantages of the proposed method in terms of accuracy and robustness in real-time operation.

Vehicle platooning is an innovative technology for intelligent transportation systems. Each vehicle in the platoon is required to autonomously follow its front vehicle's path unconditionally and accurately. For platoons without communication, accurate path and velocity estimation of the front vehicle is challenging and crucial. Instead of memorizing the original detection result of the front vehicle to generate a path, the path is estimated by fusing motion prediction and observation in this paper. Since there is a correlation between the two estimates, Split Covariance Intersection Filter is used to guarantee the fusion consistency. Besides, a motion model considering velocity is used in the filter to achieve a precise estimate of the predecessor's velocity simultaneously. Moreover, a path generation approach is designed in a high-frequency loop independent of front vehicle detection to improve continuity of the path. Experimental validations in the real-world environment highlight the remarkable improvement in the accuracy of both path and velocity estimation. Meanwhile, the path generated by the proposed approach is more continuous compared with the commonly used method. Furthermore, complete vehicle platooning demonstrations in diverse environments prove the practicality and robustness of the proposed approach.

Understanding the surrounding environment of the vehicle is still one of the challenges for autonomous driving. This paper addresses 360-degree road scene semantic segmentation using surround view cameras, which are widely equipped in existing production cars. First, in order to address large distortion problem in the fisheye images, Restricted Deformable Convolution (RDC) is proposed for semantic segmentation, which can effectively model geometric transformations by learning the shapes of convolutional filters conditioned on the input feature map. Second, in order to obtain a large-scale training set of surround view images, a novel method called zoom augmentation is proposed to transform conventional images to fisheye images. Finally, an RDC based semantic segmentation model is built; the model is trained for real-world surround view images through a multi-task learning architecture by combining real-world images with transformed images. Experiments demonstrate the effectiveness of the RDC to handle images with large distortions, and that the proposed approach shows a good performance using surround view cameras with the help of the transformed images.

Local mapping plays an important role in outdoor intelligent vehicle applications and multi-vehicle cooperative local mapping which takes advantage of vehicular communication can bring considerable benefits to this important task. In this paper, a multi-vehicle cooperative local mapping architecture using split covariance intersection filter (Split CIF) is proposed. In the proposed method, a vehicle can flexibly perform cooperative local mapping with other vehicles in decentralized way, without complicated monitoring and controlling of data flow among vehicles; fused maps can be shared freely among vehicles. An efficient and accurate implementation of the Split CIF is also introduced. A simulation-based comparative study demonstrates the potential and advantage of the proposed multi-vehicle cooperative local mapping architecture using Split CIF.

This paper presents a Graph Matching Pose SLAM to build the perception map which is a prerequisite of localization and environment perception of the intelligent vehicle. Graph-based simultaneous localization and mapping (SLAM) is widely used to build a map with global consistency and requires loop closure to eliminate the accumulative errors. However, loop closure forces the vehicle to re-visit a previously entered area. In this paper, a road network-based graph SLAM method without the requirement of loop closure is proposed to build a consistent map for the real environment. The estimated poses from the original SLAM are linked with the available road network based on the graph matching. Thus the available road network introduces the real topology of the environment to Pose SLAM regularly. In this way, the map is optimized by using the factor graph inference frequently. The proposed method is validated on the KITTI dataset and the real-world experiments. This method outperforms the state-of-the-art methods.

Terrain-based vehicle localization is a valuable alternative to GPS when the signal is blocked or dynamic occlusions exist. However, braking events can induce the slips and vibrations of the vehicle and, thus, increase the errors of terrain-based position estimations. This paper develops a terrain-based localization approach which is robust to braking events. First, the terrain map is generated and stored before localization, which includes distance measurements, terrain feature data, and the geographic locations. The terrain feature data are pitch differences to eliminate the accumulated error. Next, particle-filter-based and acceleration-considered vehicle localization is proposed for position estimation. It takes into account the influence of braking in the vehicle model and the inference process. Experimental results demonstrate the repeatability of pitch difference measurements and the robustness of the proposed approach in the cases of smooth driving and braking.

Magnetic guidance is commonly used in real applications of automated vehicles for its reliability. However, due to the downward view of magnet detectors, the control of a vehicle based on magnetic guidance is difficult, particularly in an urban environment. This paper proposes the strategy of using a curvature map of magnets to carry out look-ahead control for magnetic guidance-based vehicles. First, a Gauss curve fitting-based lateral offset estimation method is proposed in the magnet localization to improve the robustness to sensors' noise. Then, the curvature of the road is generated based on a magnet-tracking algorithm, which records passed road segments by tracking every magnet detected by the magnetic sensor. Magnet-tracking results describe the relative position of the vehicle with respect to the reference road, whereas the curvature map contains forward road information. The fusion of these two sources of information gives a full description of the local road segment. Furthermore, to suppress noises, a simple local road model with a general form is used to fuse tracking results and curvature map information. Compared with magnetic guidance methods based on accurately positioned magnets, the proposed curvature map-based method boasts a simple map-generation process, in which the vehicle only needs to be driven manually along the reference trajectory once. Experiments on real application scenarios have verified the effectiveness of the proposed methods.

Local mapping is valuable for many real-time applications of intelligent vehicle systems. Multivehiclecooperative local mappingcan bring considerable benefits to vehicles operating in some challenging scenarios. In this paper, we introduce a method of occupancy grid map merging, dedicated to multivehicle cooperative local mapping purpose in outdoor environments. In a general map merging framework, we propose an objective function based on occupancy likelihood and provide some concrete procedures designed in the spirit of genetic algorithm to optimize the defined objective function. Based on the introduced method, we further describe a strategy of indirect vehicle-to-vehicle (V2V) relative pose (RP) estimation, which can serve as a general solution for multivehicle perception association. We present a variety of experiments that validate the effectiveness of the proposed occupancy grid map merging method. We also demonstrate several useful application examples of the indirect V2V RP estimation strategy.

Data fusion is an important process in a variety of tasks in the intelligent transportation systems field. Most existing data fusion methods rely on the assumption of conditional independence or known statistics of data correlation. In contrast, the split covariance intersection filter (split CIF) was heuristically presented in literature, which aims at providing a mechanism to reasonably handle both known independent information and unknown correlated information in source data. In this paper, we provide a theoretical foundation for the split CIF. First, we clearly specify the consistency definition (coined as split consistency) for estimates in split form. Second, we provide a theoretical proof for the fusion consistency of the split CIF. Finally, we provide a theoretical derivation of the split CIF for the partial observation case. We also present a general architecture of decentralized vehicle localization, which serves as a concrete application example of the split CIF to demonstrate the advantages of the split CIF and how it can potentially benefit vehicle localization (noncooperative and cooperative). In general, this paper aims at providing a baseline for researchers who might intend to incorporate the split CIF (a useful tool for general data fusion) into their prospective research works.

Autonomous mapping, especially in the form of SLAM (Simultaneous Localization And Mapping), has long since been used for many indoor robotic applications and is also useful in outdoor intelligent vehicle applications such as object detection. Most existing research works on environment mapping and object detection in outdoor applications have been dedicated to single vehicle system. On the other hand, multi-vehicle cooperative perception based on inter-vehicle data sharing can bring considerable benefits in many scenarios that are challenging for a single vehicle system. In this paper, a new method for occupancy grid maps merging is proposed: an objective function based on occupancy likelihood is introduced to measure the consistency degree of maps alignment; genetic algorithm implemented in a dynamic scheme is adopted to optimize the objective function. A scheme of multi-vehicle cooperative local mapping and moving object detection using the proposed occupancy grid maps merging method is also introduced. Real-data tests are given to demonstrate the effectiveness of the introduced method.

Vehicle localization (ground vehicles) is an important task for intelligent vehicle systems and vehicle cooperation may bring benefits for this task. A new cooperative multi-vehicle localization method using split covariance intersection filter is proposed in this paper. In the proposed method, each vehicle maintains an estimate of a decomposed group state and this estimate is shared with neighboring vehicles; the estimate of the decomposed group state is updated with both the sensor data of the ego-vehicle and the estimates sent from other vehicles; the covariance intersection filter which yields consistent estimates even facing unknown degree of inter-estimate correlation has been used for data fusion. A comparative study based simulations demonstrate the effectiveness and the advantage of the proposed cooperative localization method.