Microscopic traffic simulation plays a crucial role in transportation engineering by providing insights into in-dividual vehicle behavior and overall traffic flow. How-ever, creating a realistic simulator that accurately repli-cates human driving behaviors in various traffic conditions presents significant challenges. Traditional simulators relying on heuristic models often fail to deliver accurate simulations due to the complexity of real-world traffic environments. Due to the covariate shift issue, existing imitation learning-based simulators often fail to generate stable long-term simulations. In this paper, we propose a novel approach called learner-aware supervised imitation learning to address the covariate shift problem in multi-agent imi-tation learning. By leveraging a variational autoencoder simultaneously modeling the expert and learner state distribution, our approach augments expert states such that the augmented state is aware of learner state distribution. Our method, applied to urban traffic simulation, demon-strates significant improvements over existing state-of-the-art baselines in both short-term microscopic and long-term macroscopic realism when evaluated on the real-world dataset pNEUMA.

Point cloud panoptic segmentation is a challenging task that seeks a holistic solution for both semantic and instance segmentation to predict groupings of coherent points. Previous approaches treat semantic and instance segmentation as surrogate tasks, and they either use clustering methods or bounding boxes to gather instance groupings with costly computation and hand-craft designs in the instance segmentation task. In this paper, we propose a simple but effective point cloud unified panoptic segmentation (PUPS) framework, which use a set of point-level classifiers to directly predict semantic and instance groupings in an end-to-end manner. To realize PUPS, we introduce bipartite matching to our training pipeline so that our classifiers are able to exclusively predict groupings of instances, getting rid of hand-crafted designs, e.g. anchors and Non-Maximum Suppression (NMS). In order to achieve better grouping results, we utilize a transformer decoder to iteratively refine the point classifiers and develop a context-aware CutMix augmentation to overcome the class imbalance problem. As a result, PUPS achieves 1st place on the leader board of SemanticKITTI panoptic segmentation task and state-of-the-art results on nuScenes.

As a key problem of auto-vehicle applications, the goal of Anomaly Obstacle Segmentation (AOS) is to detect some strange and unexpected obstacles (possibly are unseen previously) on the drivable area, thereby equipping the semantic perceptual model to be tolerant of unknown things. Due to its practicality, recently AOS is drawing attentions and a long line of works are proposed to tackle the obstacles with almost infinite diversity. However, these methods usually focus less on the priors of driving scenarios and involve image re-generation or the retraining of perceptual model, which lead to large computational quantity or the degradation of perceptual performance. In this paper, we propose to pay more attention to the characteristics of driving scenarios, lowering the difficulty of this tricky task. A training-free retrieval based method is thereby proposed to distinguish road obstacles from the surrounding road texture by computing the cosine similarity based on their appearance features, and significantly outperforms methods of the same category by around 20 percentage points. Besides, we find that there is a deep relation between our method and self-attention mechanism, and as a result a novel Transformer evolves from our retrieval based method, further boosting the performance.

LiDAR and camera are two common sensors to collect data in time for 3D object detection under the autonomous driving context. Though the complementary information across sensors and time has great potential of benefiting 3D perception, taking full advantage of sequential cross-sensor data still remains challenging. In this paper, we propose a novel LiDAR Image Fusion Transformer (LIFT) to model the mutual interaction relationship of cross-sensor data over time. LIFT learns to align the input 4D sequential cross-sensor data to achieve multi-frame multi-modal information aggregation. To alleviate computational load, we project both point clouds and images into the bird-eye-view maps to compute sparse grid-wise self-attention. LIFT also benefits from a cross-sensor and cross-time data augmentation scheme. We evaluate the proposed approach on the challenging nuScenes and Waymo datasets, where our LIFT performs well over the state-of-the-art and strong baselines.

Depth completion aims to recover the dense depth map from sparse depth data and RGB image respectively. However, due to the huge difference between the multi-modal signal input, vanilla convolutional neural network and simple fusion strategy cannot extract features from sparse data and aggregate multi-modal information effectively. To tackle this problem, we design a novel network architecture that takes full advantage of multi-modal features for depth completion. An effective Pre-completion algorithm is first put forward to increase the density of the input depth map and to provide distribution priors. Moreover, to effectively fuse the image features and the depth features, we propose a multi-modal deep aggregation block that consists of multiple connection and aggregation pathways for deeper fusion. Furthermore, based on the intuition that semantic image features are beneficial for accurate contour, we introduce the deformable guided fusion layer to guide the generation of the dense depth map. The resulting architecture, called MDANet, outperforms all the stateof-the-art methods on the popular KITTI Depth Completion Benchmark, meanwhile with fewer parameters than recent methods. The code of this work will be available at https://github.com/USTC-Keyanjie/MDANet_ICRA2021.

We present a tightly-coupled multi-sensor fusion architecture for autonomous vehicle applications, which achieves centimetre-level accuracy and high robustness in various scenarios. In order to realize robust and accurate point-cloud feature matching we propose a novel method for extracting structural, highly discriminative features from LiDAR point clouds. For high frequency motion prediction and noise propagation, we use incremental on-manifold IMU pre-integration. We also adopt a multi-frame sliding window square root inverse filter, so that the system maintains numerically stable results under the premise of limited power consumption. To verify our methodology, we test the fusion algorithm in multiple applications and platforms equipped with a LiDAR-IMU system. Our results demonstrate that our fusion framework attains state-of-the-art localization accuracy, high robustness and a good generalization ability.