Scene information plays a crucial role in trajectory forecasting systems for autonomous driving by providing semantic clues and constraints on potential future paths of traffic agents. Prevalent trajectory prediction techniques often take High-Definition maps (HD maps) as part of the inputs to provide scene knowledge. Although HD maps offer accurate road information, they may suffer from the high cost of annotation or restrictions of law, which limits their widespread use. Therefore, it is crucial for trajectory prediction methods to generate reliable prediction results in mapless scenarios. In this paper, we tackle the problem of improving the consistency of predicted trajectories and the scene road topology when map information is unavailable during the test phase. To achieve this, we propose a universal knowledge distillation (KD) framework. This KD framework trains a map-based teacher network on samples with annotated HD maps and subsequently transfers the knowledge to a student mapless predictor through a two-fold knowledge distillation process. Experimental results show that our method stably improves prediction performance in test-time mapless situations on many widely used trajectory prediction baselines, and achieves state-of-the-art mapless prediction performances. Qualitative visualization results demonstrate that our approach helps infer unseen map information. Our solution is generalizable for common trajectory prediction networks and datasets.

Multi-agent trajectory prediction (MATP) is pivotal in both robotics and autonomous driving. Current multi-agent trajectory prediction methodologies face significant performance degradation due to distributional discrepancies between pre-training data and real-world deployment environments. In practical scenarios, data arrive as a continuous data stream with distribution shifts and potential recurrence of older patterns, constituting a continual learning trajectory prediction problem. Training on a continuous data stream is difficult due to the evolving domain knowledge regarding both intra-sample interactions and trajectory statistical patterns, leading to catastrophic forgetting. We tackle this challenge by proposing a rehearsal-based dual knowledge consolidation framework, which systematically consolidates interaction patterns via intra-sample relational distillation, and retains trajectory statistics via an expandable pattern memory bank and knowledge distillation between pattern centers and features. A pseudo-data generation strategy is also newly proposed to enhance the rehearsal diversity. We design challenging yet practical continual prediction experiments on widely used datasets to evaluate our method. Results demonstrate that our approach significantly improves the prediction performance on continuous data, surpassing existing continuous prediction methods.

Predicting the future trajectories of the traffic agents is a gordian technique in autonomous driving. However, trajectory prediction suffers from data imbalance in the prevalent datasets, and the tailed data is often more complicated and safety-critical. In this paper, we focus on dealing with the long-tail phenomenon in trajectory prediction. Previous methods dealing with long-tail data did not take into account the variety of motion patterns in the tailed data. In this paper, we put forward a future enhanced contrastive learning framework to recognize tail trajectory patterns and form a feature space with separate pattern clusters. Furthermore, a distribution aware hyper predictor is brought up to better utilize the shaped feature space. Our method is a model-agnostic framework and can be plugged into many well-known baselines. Experimental results show that our framework outperforms the state-of-the-art long-tail prediction method on tailed samples by 9.5% on ADE and 8.5% on FDE, while maintaining or slightly improving the averaged performance. Our method also surpasses many long-tail techniques on trajectory prediction task.

LiDAR-Camera online calibration is of great significance for building a stable autonomous driving perception system. For online calibration, a key challenge lies in constructing a unified and robust representation between multi-modal sensor data. Most methods extract features manually or implicitly with an end-to-end deep learning method. The former suffers poor robustness, while the latter has poor interpretability. In this paper, we propose CalibDepth, which uses depth maps as the unified representation for image and LiDAR point cloud. CalibDepth introduces a sub-network for monocular depth estimation to assist online calibration tasks. To further improve the performance, we regard online calibration as a sequence prediction problem, and introduce global and local losses to optimize the calibration results. CalibDepth shows excellent performance in different experimental setups. Code is open-sourced at https://github.com/Brickzhuantou/CalibDepth.

Trajectory forecasting for traffic participants (e.g., vehicles) is critical for autonomous platforms to make safe plans. Currently, most trajectory forecasting methods assume that object trajectories have been extracted and directly develop trajectory predictors based on the ground truth trajectories. However, this assumption does not hold in practical situations. Trajectories obtained from object detection and tracking are inevitably noisy, which could cause serious forecasting errors to predictors built on ground truth trajectories. In this paper, we propose to predict trajectories directly based on detection results without relying on explicitly formed trajectories. Different from traditional methods which encode the motion cues of an agent based on its clearly defined trajectory, we extract the motion information only based on the affinity cues among detection results, in which an affinity-aware state update mechanism is designed to manage the state information. In addition, considering that there could be multiple plausible matching candidates, we aggregate the states of them. These designs take the uncertainty of association into account which relax the undesirable effect of noisy trajectory obtained from data association and improve the robustness of the predictor. Extensive experiments validate the effectiveness of our method and its generalization ability to different detectors or forecasting schemes.

Heterogeneous trajectory forecasting is critical for intelligent transportation systems, but it is challenging because of the difficulty of modeling the complex interaction relations among the heterogeneous road agents as well as their agent-environment constraints. In this work, we propose a risk and scene graph learning method for trajectory forecasting of heterogeneous road agents, which consists of a Heterogeneous Risk Graph (HRG) and a Hierarchical Scene Graph (HSG) from the aspects of agent category and their movable semantic regions. HRG groups each kind of road agent and calculates their interaction adjacency matrix based on an effective collision risk metric. HSG of the driving scene is modeled by inferring the relationship between road agents and road semantic layout aligned by the road scene grammar. Based on this formulation, we can obtain effective trajectory forecasting in driving situations, and comparable performance to other state-of-the-art approaches is presented by extensive experiments on the nuScenes, ApolloScape, and Argoverse datasets.

In the task of pedestrian trajectory prediction, social interaction could be one of the most complicated factors since it is difficult to be interpreted through simple rules. Recent studies have shown a great ability of LSTM networks in learning social behaviors from datasets, e.g., introducing LSTM hidden states of the neighbors at the last time step into LSTM recursion. However, those methods depend on previous neighboring features which lead to a delayed observation. In this paper, we propose a data-driven states refinement LSTM network (SR-LSTM) to enable the utilization of the current intention of neighbors through a message passing framework. Moreover, the model performs in the form of self-updating by jointly refining the current states of all participants, rather than an input-output mechanism served by feature concatenation. In the process of states refinement, a social-aware information selection module consisting of an element-wise motion gate and a pedestrian-wise attention is designed to serve as the guidance of the message passing process. Considering the pedestrian walking space as a graph where each pedestrian is a node and each pedestrian pair with an edge, spatial-edge LSTMs are further exploited to enhance the model capacity, where two kinds of LSTMs interact with each other so that states of them are interactively refined. Experimental results on four widely used pedestrian trajectory datasets, ETH, UCY, PWPD, and NYGC demonstrate the effectiveness of the proposed model.

In crowd scenarios, reliable trajectory prediction of pedestrians requires insightful understanding of their social behaviors. These behaviors have been well investigated by plenty of studies, while it is hard to be fully expressed by hand-craft rules. Recent studies based on LSTM networks have shown great ability to learn social behaviors. However, many of these methods rely on previous neighboring hidden states but ignore the important current intention of the neighbors. In order to address this issue, we propose a data-driven state refinement module for LSTM network (SR-LSTM), which activates the utilization of the current intention of neighbors, and jointly and iteratively refines the current states of all participants in the crowd through a message passing mechanism. To effectively extract the social effect of neighbors, we further introduce a social-aware information selection mechanism consisting of an element-wise motion gate and a pedestrian-wise attention to select useful message from neighboring pedestrians. Experimental results on two public datasets, i.e. ETH and UCY, demonstrate the effectiveness of our proposed SR-LSTM and we achieve state-of-the-art results.

In autonomous driving community, numerous benchmarks have been established to assist the tasks of 3D/2D object detection, stereo vision, semantic/instance segmentation. However, the more meaningful dynamic evolution of the surrounding objects of ego-vehicle is rarely exploited, and lacks a large-scale dataset platform. To address this, we introduce BLVD, a large-scale 5D semantics benchmark which does not concentrate on the static detection or semantic/instance segmentation tasks tackled adequately before. Instead, BLVD aims to provide a platform for the tasks of dynamic 4D (3D+temporal) tracking, 5D (4D+interactive) interactive event recognition and intention prediction. This benchmark will boost the deeper understanding of traffic scenes than ever before. We totally yield 249, 129 3D annotations, 4, 902 independent individuals for tracking with the length of overall 214, 922 points, 6, 004 valid fragments for 5D interactive event recognition, and 4, 900 individuals for 5D intention prediction. These tasks are contained in four kinds of scenarios depending on the object density (low and high) and light conditions (daytime and nighttime). The benchmark can be downloaded from our project site https://github.com/VCCIV/BLVD/.