Context: Simulation-based testing is a cost-efficient alternative to field testing for Autonomous Vehicles (AVs), but generating safety-critical test cases is challenging due to the vast search space. Prior work has studied static (road features) and dynamic (AV behavior) features of test scenarios separately, but their inter-dependencies are underexplored. Objective: In this paper, we describe an empirical to analyze how static and dynamic features of test scenarios, and their inter-dependencies, influence AV test scenario outcomes. Method: This study proposes an integrated approach using Instance Space Analysis (ISA) to evaluate both types of features, identify key influences on AV safety, and predict test outcomes without execution. Results: Our study identifies critical features affecting test outcomes (effective/ineffective, depending on whether it leads to a safety-critical condition). Results show that combining static and dynamic features improves prediction accuracy, confirmed by models trained on both feature types outperforming models trained with only one type of feature. Conclusion: The interplay of static and dynamic features enhances fault detection in AV testing. This research underscores the importance of integrating both types of features to create more effective testing frameworks for autonomous systems. Key contributions include: (1) a unified framework for AV safety assessment, (2) identification of influential features using ISA, and (3) efficient test outcome prediction for optimized regression testing.

Machine learning (ML) has shown promising results in detecting software vulnerabilities. However, ML detectors are not guaranteed to make predictions based on the right indicators. Studies have revealed that they can rely onirrelevantcode features, such as identifiers or function signatures, particularly those that commonly appear in vulnerable code, yet are not related to the actual vulnerabilities. As a result, the lines of code that the detectors depend on and flag as suspicious are not always genuinely vulnerable. Consequently, developers must manually review these suspicious lines, which is time-consuming and error-prone. If the suspicious lines are wrong, developers may be misled, spend unnecessary effort, or even reach incorrect patching strategies. This highlights the need for automated approaches to identify untrustworthy vulnerability predictions.In this paper, we introduce UNTRUSTVUL, a new approach for identifying untrustworthy vulnerability predictions. Specifically, we focus on cases where a model highlights suspicious lines that would not appear in reliable predictions, i.e., lines that are inherently non-vulnerable and unrelated to any vulnerabilities. To achieve this, we leverage patterns of vulnerable lines observed in historical data. UNTRUSTVUL automatically rules out as untrustworthy any predictions that highlight suspicious lines neither observed in history nor influential to those that have been observed. We refer to such lines as vulnerability-irrelevant. A line is deemed vulnerability-irrelevant if ① it does not match any known patterns of historical vulnerabilities, and ② all its successors in the data and control dependency graph are also vulnerability-irrelevant. Intuitively, a vulnerability-irrelevant line shows low similarity to known vulnerabilities and has no dependency paths to any lines outside the vulnerability-irrelevant category. Notably, these rules are designed to be conservative, as mislabeling a trustworthy prediction as untrustworthy is also undesired. We evaluate UNTRUSTVULon 115K vulnerability predictions made by four models across BigVul, MegaVul, SARD, and PrimeVul datasets, with ground-truth trustworthiness labeled based on the overlap between actual denoised vulnerable lines and model-annotated suspicious lines. UNTRUSTVULeffectively detects untrustworthy predictions with AUC of 70%–88% and F1–score of 82%–94%, outperforming existing approaches by 6%–59% in AUC and 13%–92% in F1–score.

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a ( \(2D\) ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

Autonomous Vehicles (AVs) are prone to revolutionise the transportation industry. However, they must be thoroughly tested to avoid safety violations. Simulation testing plays a crucial role in finding safety violations of Automated Driving Systems (ADSs). This paper proposes PAFOT, a position-based approach testing framework, which generates adversarial driving scenarios to expose safety violations of ADSs. We introduce a 9-position grid which is virtually drawn around the Ego Vehicle (EV) and modify the driving behaviours of Non-Playable Characters (NPCs) to move within this grid. PAFOT utilises a single-objective genetic algorithm to search for adversarial test scenarios. We demonstrate PAFOT on a well-known high-fidelity simulator, CARLA. The experimental results show that PAFOT can effectively generate safety-critical scenarios to crash ADSs and is able to find collisions in a short simulation time. Furthermore, it outperforms other search-based testing techniques by finding more safety-critical scenarios under the same driving conditions within less effective simulation time.

AI-powered systems have gained widespread popularity in various domains, including Autonomous Vehicles (AVs). However, ensuring their reliability and safety is challenging due to their complex nature. Conventional test adequacy metrics, designed to evaluate the effectiveness of traditional software testing, are often insufficient or impractical for these systems. White-box metrics, which are specifically designed for these systems, leverage neuron coverage information. These coverage metrics necessitate access to the underlying AI model and training data, which may not always be available. Furthermore, the existing adequacy metrics exhibit weak correlations with the ability to detect faults in the generated test suite, creating a gap that we aim to bridge in this study.

Ensuring the safety of autonomous vehicles (AVs) is of utmost importance, and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive, as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road’s structural features (lanes, turns, road speed, etc.). In this article, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in two dimensions. This visualisation helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritisation.

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances – in the form of Instance Space Analysis (ISA) – enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

Summary Debugging is crucial for producing reliable software. One of the effective bug localization techniques is spectral‐based fault localization. It tries to locate a buggy statement by applying an evaluation metric to program spectra and ranking program components on the basis of the score it computes. Here, we propose a restricted class of “hyperbolic” metrics, with a small number of numeric parameters. This class of functions is based on past theoretical and empirical results. We show that optimization methods such as genetic programming and simulated annealing can reliably discover effective metrics over a wide range of data sets of program spectra. We evaluate the performance for both real programs and model programs with single bugs, multiple bugs, “deterministic” bugs, and nondeterministic bugs and find that the proposed class of metrics performs as well as or better than the previous best‐performing metrics over a broad range of data.

Summary Debugging is crucial for producing reliable software. One of the effective bug localization techniques is spectral‐based fault localization (SBFL). It helps to locate a buggy statement by applying an evaluation metric to program spectra and ranking program components on the basis of the score it computes. SBFL is an example of a dynamic analysis – an analysis of computer program that is performed by executing it with sufficient number of test cases. Static analysis, on the other hand, is performed in a non‐runtime environment. We introduce a weighting technique by combining these two kinds of program analysis. Static analysis is performed to categorize program statements into different classes and giving them weights based on the likelihood of being buggy statement. Statements are finally ranked on the basis of the weights computed by statements' categorization (static analysis) and scores computed by SBFL metrics (dynamic analysis). We evaluate the performance of our technique on Siemens test suite and Flex (having seeded bugs seeded by expert developers), Sed (having mixture of real and seeded bugs), and Space (having real bugs). In our evaluation, proposed weighting technique improves the performance of a wide variety of fault localization metrics up to 20% on single bug datasets and up to 42% on multi‐bug datasets. Copyright © 2017 John Wiley & Sons, Ltd.

Open source projects for example Eclipse and Fire fox have open source bug repositories. User reports bugs to these repositories. Users of these repositories are usually non-technical and cannot assign correct class to these bugs. Triaging of bugs, to developer, to fix them is a tedious and time consuming task. Developers are usually expert in particular areas. For example, few developers are expert in GUI and others are in java functionality. Assigning a particular bug to relevant developer could save time and would help to maintain the interest level of developers by assigning bugs according to their interest. However, assigning right bug to right developer is quite difficult for triager without knowing the actual class, the bug belongs to. In this research, we have classified the bugs in different labels on the basis of summary of the bug. Multinomial Naïve Bayes text classifier is used for classification purpose. For feature selection, Chi-Square and TFIDF algorithms were used. Using Naïve Bayes and Chi-square, we get average of 83 % accuracy.