Critical scenario generation plays a crucial role in the autonomous driving test by efficiently and effectively identifying various hazardous scenarios to evaluate the multiagent system under test. The performance of existing solution models is hampered by sparse rewards resulting from long time-steps in driving scenarios. Moreover, they fail to guide the generation of more diverse scenarios because of the lack of a fine-grained design. To efficiently and effectively discover various critical scenarios, we propose the MACS method based on multiagent reinforcement learning to guide adversaries foiled the agent under test by replay buffer optimization and objective function design. By adopting the hindsight experience replay method, historical experiences are reused to address the challenge of sparse rewards and improve sample efficiency. Furthermore, we integrate the entropy term into the objective function to explore different driving strategies, thereby leading to the creation of diverse scenarios. We have achieved a new state-of-the-art performance in evaluating rule-based agents using an industrial-grade platform, SMARTS. The experimental results demonstrate that MACS can effectively generate diverse critical scenarios that lead to the failure of the agent under test. We also apply cluster methods, including DBSCAN and TRACLUS, to conduct diversity analysis of the generated scenarios. Besides, we evaluate and improve the reinforcement learning decision algorithm for the vehicle under test with our generated scenarios and give empirical conclusions about its robustness.

Exploring critical scenarios for autonomous driving is a challenging task that requires effectively generating diverse multimodal scenarios from an infinite parameter space. However, most search-based methods encounter the mode collapse problem, leading to the generation of similar concrete scenarios for a given logical scenario. Besides, they require training different models for potentially diverse logical scenarios, which consume considerable time. To tackle these challenges, we propose the Stacked Action Blocks (SAB) framework inspired by reusable modularity in software architecture. In this framework, we extract atomic actions from logical scenarios and train each atomic action using a multimodal model as an action block to mitigate the impact of mode collapse. By reusing these trained action blocks, we compose diverse critical scenarios, thereby reducing training costs. We extensively evaluated our approach in four complex driving scenarios and achieved superior performance in the Critical Scenario Generation (CSG) task compared to heuristic methods and search-based methods. We demonstrate that the independent training of these action blocks that are reusable and modular is an effective way to find diverse multimodal critical driving scenarios. In addition, we show that the driving policy becomes better at avoiding collisions after fine-tuning based on these critical scenarios.

Rare critical scenarios are crucial to verify the performance of autonomous driving in different situations. Critical scenario generation requires the ability of sampling critical combinations from an infinite parameter space in the logical scenario. Existing solutions aim to explore the correlation of action parameters in the initial scenario rather than action sequences. How to model action sequences so that one can further consider the effects of different action parameters is the bottleneck of the problem. In this paper, we solve the problem by proposing the ECSAS framework. Specifically, we first propose a description language, BTScenario, allowing us to model action sequences of scenarios. We then use reinforcement learning to search for combinations of critical action parameters. Several optimizations are proposed to increase efficiency, including action mask and replay buffer. Experimental results show that our model with strong collision ability and effectively outperforms the existing methods on various nontrivial scenarios.