Long-tail scenarios are of great significance in improving the testing efficacy of autonomous vehicles, but they are difficult to collect due to their inherent randomness and high risk. The low-cost, high-value videos of vehicle-perspective accidents stored in tachographs provide a potentially important source for long-tail data collection. However, the lack of multisource synchronization information and the unknown, diverse camera parameters in tachograph videos make it difficult for the existing methods to accurately extract key information. In response to these issues, this work proposes a vehicle-perspective accident scenario restoration framework based on tachograph videos. First, using preprocessed accident videos as the input, a scenario semantic understanding module was constructed to help extract the pixel trajectories, conduct parameter calibration, and obtain pixel road boundaries. Then, robust detection and tracking algorithms were used to extract the pixel trajectories. Furthermore, a high-precision calibration algorithm was proposed based on the Geometrically Salient Feature Bundle Adjustment (GSFBA). Finally, the camera parameters were used to perform an inverse perspective transformation on the pixel coordinates, the actual road boundaries were extracted based on the mapping strength of the pixel road boundaries, and the trajectory information was optimized using a systematic trajectory processing module. The proposed method was verified at multiple levels based on both two simulation scenarios and two real-world scenarios. The results show that the proposed framework in this work can obtain the key information of the scenario with a high degree of restoration accuracy. It was effectively applied to the restoration process of accident scenarios.

Accident scenarios with long-tail characteristics are essential for advancing autonomous vehicles (AVs) functionality. Integrating such scenarios into the training process enhances adaptability to complex situations. However, the scarcity of accident data, due to their randomness and collection challenges, limits this integration. To address this issue, a framework named CRADLE is proposed, leveraging causal reinforcement learning (RL) and deep learning for accident scenario generation. Under extremely limited data conditions, CRADLE enables the construction of a highly diverse accident scenario database while ensuring consistency in accident causation. First, a scenario knowledge graph is constructed, incorporating both a scenario graph and an accident causation graph. Then, a scenario graph temporal prediction model trained on real driving data, generates a sampling space for scenario graphs. Subsequently, a causal inference module is developed to establish a mapping and transformation mechanism between temporal scenario graphs and accident causation graphs. Finally, a graph similarity measurement method is introduced to guide RL model in orderly sampling within the scenario graph sampling space, ensuring causally controlled scenario generation. The proposed method is applied to generate accident scenarios for two lane-changing situations: simultaneous lane changes and multi-lane changes. These scenarios are incorporated into the closed-loop self-evolution process of the autonomous driving algorithms. Experimental results demonstrate that the constructed accident scenario databases significantly improve the algorithm adaptability, reducing AV collision rates by approximately 76%.

Hierarchical structure is a common characteristic for some kinds of videos (e.g., sports videos, game videos): The videos are composed of several actions hierarchically and there exist temporal dependencies among segments with different scales, where action labels can be enumerated. Our ideas are based on two observations: First, the actions are the fundamental units for people to understand these videos. Second, the humans summarize a video by iteratively observing and refining, i.e., observing segments in video and hierarchically refining the boundaries of important actions. Based on the above insights, we generate action proposals to construct the structure of the video and formulate the summarization process as a hierarchical refining process. We also train a hierarchical summarization network with deep Q-learning (HQSN) to achieve the refining process and explore temporal dependency. Besides, we collect a new dataset that consists of structured game videos with fine-grain actions and importance annotations. The experimental results demonstrate the effectiveness of the proposed method.