PALM is a generator of scenarios for UAV testing, that participated in the ICST Tool Competition 2025 - CPS-UAV Test Case Generation Track. PALM adopts Monte Carlo Tree Search (MCTS) to search for different placements of obstacles of different sizes in the mission environment. By increasing the tree depth, a new obstacle is added to the environment; instead, by adding a new node in the current tree level, the tool optimises the placement and the dimension of the last added obstacle.

Autonomous Driving Systems (ADS) are considered safety-critical, as even a minor fault may lead to catastrophic consequences. To evaluate their reliability and robustness under failure conditions, Fault Injection (FI) techniques have been widely adopted. Most existing FI methods employ data-driven approaches, such as surrogate modeling and reinforcement learning, to generate test cases. While these techniques have shown promise, they often incur substantial costs in terms of data collection and training time. Moreover, their performance is highly sensitive to the quality and quantity of training data, which can limit their applicability in diverse or unseen scenarios. In this paper, we propose LOFT, an efficient multi-objective search-based FI testing framework that leverages Large Language Models (LLMs) to identify diverse and realistic critical faults. To accommodate the structured and non-linguistic nature of raw simulation data, LOFT adopts a two-stage LLM-based fault injection pipeline. In the first stage, an LLM converts singleframe simulation data into natural language descriptions and suggests appropriate fault types. In the second stage, a separate LLM examines the broader scenario context to determine the optimal time window for fault injection. The outputs from the two LLMs are then used to initialize and guide a multi-objective search procedure aiming at discovering a diverse set of critical faults. We implement LOFT and evaluate on an ADS provided by our industrial partner. Experimental results show that, compared with two baseline approaches, LOFT detects over $90 \%$ more critical faults and identifies an average of 2.2 additional fault types within an equivalent number of simulations.

Autonomous driving systems (ADS) are safety-critical and require comprehensive testing before their deployment on public roads. While existing testing approaches primarily aim at the criticality of scenarios, they often overlook the diversity of the generated scenarios that is also important to reflect system defects in different aspects. To bridge the gap, we propose LeGEND, that features a top-down fashion of scenario generation: it starts with abstract functional scenarios, and then steps downwards to logical and concrete scenarios, such that scenario diversity can be controlled at the functional level. However, unlike logical scenarios that can be formally described, functional scenarios are often documented in natural languages (e.g., accident reports) and thus cannot be precisely parsed and processed by computers. To tackle that issue, LeGEND leverages the recent advances of large language models (LLMs) to transform textual functional scenarios to formal logical scenarios. To mitigate the distraction of useless information in functional scenario description, we devise a two-phase transformation that features the use of an intermediate language; consequently, we adopt two LLMs in LeGEND, one for extracting information from functional scenarios, the other for converting the extracted information to formal logical scenarios. We experimentally evaluate LeGEND on Apollo, an industry-grade ADS from Baidu. Evaluation results show that LeGEND can effectively identify critical scenarios, and compared to baseline approaches, LeGEND exhibits evident superiority in diversity of generated scenarios. Moreover, we also demonstrate the advantages of our two-phase transformation framework, and the accuracy of the adopted LLMs.

Autonomous Driving Systems (ADS) are safety-critical and require comprehensive testing before their deployment on public roads. Most existing testing approaches consist in generating scenarios that vary the behaviors of dynamic objects, while leaving a predefined road environment unchanged. Consequently, these approaches overlook the influence of different road structures on ADS safety, e.g., collisions can happen more frequently than usual on a merging road, because of the specific road structure. In this paper, we propose EvoScenario, a novel approach that integrates road structures into the generation of critical scenarios for exposing safety risks of ADS. Specifically, EvoScenario models a driving road as a sequence of road segments characterized in different aspects, such as their shapes and widths. Then, a test case is defined by concatenating the sequence of road segments and the sequence of dynamic object maneuvers. Inspired by EvoSuite that generates sequential method calls for Java unit testing, EvoScenario leverages the sequential models of test cases and constructs a multi-objective optimization framework to search for critical scenarios. We implement and demonstrate EvoScenario on an ADS provided by our industrial partner. Evaluation results show that EvoScenario can identify 6 types of safety violations, and outperform existing baseline testing approaches.

Autonomous driving systems (ADS) are safety-critical systems that require thorough testing to ensure their safety. Current testing methods for ADS primarily focus on finding crash scenarios involving ADS. However, most of these scenarios are unavoidable by ADS, such as collisions caused by the reckless behavior of other vehicles. To address this limitation, we propose CollVer, a framework designed to generate and identify scenarios in which ADS violate driving rules. Specifically, CollVer utilizes multi-modal technology by taking the violation scenario and the corresponding accident description as inputs to judge whether the accident can be attributed to the ADS. Moreover, CollVer introduces a metric called collision position coverage (CPC), to quantify and guide the selection of test cases. Finally, CollVer integrates the multi-modal model and the CPC metric into a multi-objective genetic algorithm to explore more diverse and challenging scenarios. We evaluate CollVer on an industrial-grade ADS, Baidu Apollo, and experimental results show that CollVer can identify 10 distinct types of safety violations, with 4 of them resulting from ADS violating driving rules.

Automated Driving Systems ( ADS ) have made great achievements in recent years thanks to the efforts from both academia and industry. A typical ADS is composed of multiple modules, including sensing, perception, planning, and control, which brings together the latest advances in different domains. Despite these achievements, safety assurance of ADS is of great significance, since unsafe behavior of ADS can bring catastrophic consequences. Testing has been recognized as an important system validation approach that aims to expose unsafe system behavior; however, in the context of ADS, it is extremely challenging to devise effective testing techniques, due to the high complexity and multidisciplinarity of the systems. There has been great much literature that focuses on the testing of ADS, and a number of surveys have also emerged to summarize the technical advances. Most of the surveys focus on the system-level testing performed within software simulators, and they thereby ignore the distinct features of different modules. In this article, we provide a comprehensive survey on the existing ADS testing literature, which takes into account both module-level and system-level testing. Specifically, we make the following contributions: (1) We survey the module-level testing techniques for ADS and highlight the technical differences affected by the features of different modules; (2) we also survey the system-level testing techniques, with focuses on the empirical studies that summarize the issues occurring in system development or deployment, the problems due to the collaborations between different modules, and the gap between ADS testing in simulators and the real world; and (3) we identify the challenges and opportunities in ADS testing, which pave the path to the future research in this field.