This paper addresses the need for an explainable and generalized intrusion detection system (IDS) for the in-vehicle networks (IVNs). While machine learning (ML)-based IDS solutions show promising performance, there are still some challenges, such as the lack of trustworthiness and scarcity of attack representing data, hindering their adoption in the automotive cybersecurity. To address these issues, this paper proposes a centralized ML model training and decentralized execution-based framework, namely ACHILLES, that facilitates an explainable and generalizable automotive IDS. Under ACHILLES, different ML models can be trained centrally to enhance decentralized and onboard intrusion detection performance with multiple automotive datasets. In addition, we generate standard feature formats to assess the ML model’s generalization efficacy, where the quality of generalization and explainability is evaluated with SHapley Additive exPlanations (SHAP) by identifying the importance of the feature. We also propose a meta-learning scheme to construct suitable ML models trained by the proposed standard feature formats. The proposed feature format exhibits significant performance gain during ML model training and testing with four state-of-the-art controller area network (CAN)-bus datasets containing real, advanced attacks. The experimental results indicate that developing ML models using the generated generalized features and the meta learning-based model building process leads to enhanced performance. In particular, under the dataset cross train-test setting, the proposed feature format enhances the average accuracy by 40.1% for the baseline model, 32.4% for the meta-learned DNN, and 23.6% for the meta-learned Random Forest, compared with the baseline feature format.

Context and Motivation] The digital evolution of connected vehicles and the subsequent security risks emphasize the critical need for implementing in-vehicle cyber security measures such as intrusion detection and response systems. The continuous advancement of attack scenarios further highlights the need for adaptive detection mechanisms that can detect evolving, unknown, and complex threats. The effective use of ML-driven techniques can help address this challenge. [Problem] However, constraints on implementing diverse attack scenarios on test vehicles due to safety, cost, and ethical considerations result in a scarcity of data representing attack scenarios. This limitation necessitates alternative efficient and effective methods for generating high-quality attack-representing data. [Contribution] This paper presents a context-aware attack data generator that generates attack inputs and corresponding invehicle network log, i.e., controller area network (CAN) log, representing various types of attack including denial of service (DoS), fuzzy, spoofing, suspension, and replay attacks. It utilizes parameterized attack models augmented with CAN message decoding and attack intensity adjustments to configure the attack scenarios with high similarity to real-world scenarios and promote variability. We evaluate the practicality of the generated attack-representing data within an intrusion detection system (IDS) case study, in which we develop and perform an empirical evaluation of two deep neural network IDS models using the generated data. In addition to the efficiency and scalability of the approach, the performance results of IDS models, high detection and classification capabilities, validate the consistency and effectiveness of the generated data as well.

Using machine learning to detect and identify cyberattacks in Industrial Control Systems (ICS) offers a promising solution for uncovering zero-day attacks that traditional rulebased models cannot detect. However, applying ML-based intrusion detection in ICS environments presents challenges, including limited availability of attack data and difficulty in accurately identifying attack types. This paper addresses these challenges by proposing two key strategies. First, we demonstrate that the predictable traffic patterns of ICS networks enable the use of semi-supervised learning models for attack detection. We validate this approach using a benchmark dataset, showing that semi-supervised models achieve comparable performance to fully supervised models while relying solely on training with normal network data. Second, we propose a sequence-based approach for attack identification, using temporal data to improve the accuracy of identifying specific attack types. Our experiments reveal that incorporating historical network parameters improves the attack identification. Our research underscores the potential of semisupervised learning for effective attack detection and highlights the importance of incorporating network temporal properties to improve attack identification.

This paper presents a cybersecurity solution designed to fortify Industrial Control Systems (ICS) against cyberattacks. The proposed solution integrates a Network-based Intrusion Detection System (NIDS) with a Decision Support System (DSS), leveraging machine learning to detect anomalies in network data and employing a filtering mechanism to reduce false alarms. The NIDS protects a simulated ICS testbed, detecting anomalies and forwarding them to the DSS for further analysis and selection of mitigation strategies. We outline the system architecture and showcase promising outcomes from a prototype implementation. Our proof of concept evaluation demonstrates high accuracy in detecting attack scenarios. Challenges such as detection delays between attacks and potential mitigations high-light areas for future improvement. This research contributes to bridging the gap between ML-based IDS and security solutions, paving the way for enhanced cybersecurity in ICS environments.

Forecasting State of Health (SOH) is vital for predictive maintenance and decision-making regarding battery replacement and second-life applications. In this study, we present a high-accuracy deep learning-based SOH forecasting approach that utilizes a feature transformation technique generating histogram-based stressor features-which represent the time in which the battery cells spend under operational conditions-and augments the output with SHapley Additive exPlanations (SHAP values). We also investigate the practicality of a lightweight surrogate model, e.g., a support vector regression (SVR) model, to compare against deep neural network (DNN) models for scenarios with constrained computational resources (edge-based deployment). Our results indicate that a feature refinement utilizing Spearman rank correlation guided by the SHAP feature importance analysis to ensure coverage of critical stressor features can enable the SVR model to provide performance comparable to the DNN models.

Summary This paper presents an extended version of Deeper, a search‐based simulation‐integrated test solution that generates failure‐revealing test scenarios for testing a deep neural network‐based lane‐keeping system. In the newly proposed version, we utilize a new set of bio‐inspired search algorithms, genetic algorithm (GA), and evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain‐specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber‐physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure‐revealing test scenarios for testing an ML‐driven lane‐keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

Integration of machine learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component in an open automotive system. This paper reports results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian automatic emergency braking demonstrator running in an industry-grade simulator. We demonstrate an application of AMLAS on SMIRK for a minimalistic operational design domain, i.e., we share a complete safety case for its integrated ML-based component. Finally, we report lessons learned and provide both SMIRK and the safety case under an open-source license for the research community to reuse.

Abstract Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models.

Generating tests that can reveal performance issues in large and complex software systems within a reasonable amount of time is a challenging task. On one hand, there are numerous combinations of input data values to explore. On the other hand, we have a limited test budget to execute tests. What makes this task even more difficult is the lack of access to source code and the internal details of these systems. In this paper, we present an automated test generation method called ACTA for black-box performance testing. ACTA is based on active learning, which means that it does not require a large set of historical test data to learn about the performance characteristics of the system under test. Instead, it dynamically chooses the tests to execute using uncertainty sampling. ACTA relies on a conditional variant of generative adversarial networks, and facilitates specifying performance requirements in terms of conditions and generating tests that address those conditions. We have evaluated ACTA on a benchmark web application, and the experimental results indicate that this method is comparable with random testing, and two other machine learning methods, i.e. PerfXRL and DN.

Performance testing with the aim of generating an efficient and effective workload to identify performance issues is challenging. Many of the automated approaches mainly rely on analyzing system models, source code, or extracting the usage pattern of the system during the execution. However, such information and artifacts are not always available. Moreover, all the transactions within a generated workload do not impact the performance of the system the same way, a finely tuned workload could accomplish the test objective in an efficient way. Model-free reinforcement learning is widely used for finding the optimal behavior to accomplish an objective in many decision-making problems without relying on a model of the system. This paper proposes that if the optimal policy (way) for generating test workload to meet a test objective can be learned by a test agent, then efficient test automation would be possible without relying on system models or source code. We present a self-adaptive reinforcement learning-driven load testing agent, RELOAD, that learns the optimal policy for test workload generation and generates an effective workload efficiently to meet the test objective. Once the agent learns the optimal policy, it can reuse the learned policy in subsequent testing activities. Our experiments show that the proposed intelligent load test agent can accomplish the test objective with lower test cost compared to common load testing procedures, and results in higher test efficiency.

Differential evolution (DE) is an effective population-based metaheuristic algorithm for solving complex optimisation problems. However, the performance of DE is sensitive to the mutation operator. In this paper, we propose a novel DE algorithm, Clu-DE, that improves the efficacy of DE using a novel clustering-based mutation operator. First, we find, using a clustering algorithm, a winner cluster in search space and select the best candidate solution in this cluster as the base vector in the mutation operator. Then, an updating scheme is introduced to include new candidate solutions in the current population. Experimental results on CEC-2017 benchmark functions with dimensionalities of 30, 50 and 100 confirm that Clu-DE yields improved performance compared to DE.

Performance testing remains a challenge, particularly for complex systems. Different application-, platform- and workload-based factors can influence the performance of software under test. Common approaches for generating platform- and workload-based test conditions are often based on system model or source code analysis, real usage modeling and use-case based design techniques. Nonetheless, creating a detailed performance model is often difficult, and also those artifacts might not be always available during the testing. On the other hand, test automation solutions such as automated test case generation can enable effort and cost reduction with the potential to improve the intended test criteria coverage. Furthermore, if the optimal way (policy) to generate test cases can be learnt by testing system, then the learnt policy can be reused in further testing situations such as testing variants, evolved versions of software, and different testing scenarios. This capability can lead to additional cost and computation time saving in the testing process. In this research, we present an autonomous performance testing framework which uses a model-free reinforcement learning augmented by fuzzy logic and self-adaptive strategies. It is able to learn the optimal policy to generate platform- and workload-based test conditions which result in meeting the intended testing objective without access to system model and source code. The use of fuzzy logic and self-adaptive strategy helps to tackle the issue of uncertainty and improve the accuracy and adaptivity of the proposed learning. Our evaluation experiments show that the proposed autonomous performance testing framework is able to generate the test conditions efficiently and in a way adaptive to varying testing situations.

Automated testing activities like automated test case generation imply a reduction in human effort and cost, with the potential to impact the test coverage positively. If the optimal policy, i.e., the course of actions adopted, for performing the intended test activity could be learnt by the testing system, i.e., a smart tester agent, then the learnt policy could be reused in analogous situations which leads to even more efficiency in terms of required efforts. Performance testing under stress execution conditions, i.e., stress testing, which involves providing extreme test conditions to find the performance breaking points, remains a challenge, particularly for complex software systems. Some common approaches for generating stress test conditions are based on source code or system model analysis, or use-case based design approaches. However, source code or precise system models might not be easily available for testing. Moreover, drawing a precise performance model is often difficult, particularly for complex systems. In this research, I have used model-free reinforcement learning to build a self-adaptive autonomous stress testing framework which is able to learn the optimal policy for stress test case generation without having a model of the system under test. The conducted experimental analysis shows that the proposed smart framework is able to generate the stress test conditions for different software systems efficiently and adaptively without access to performance models.