Security Chaos Engineering (SCE) is a proactive approach to identify vulnerabilities and enhance security of systems. It embraces continuous security experimentation to build confidence in the capability of systems to withstand malicious conditions. Different SCE techniques are proposed for enhancing the resilience of software systems. The diversity of SCE techniques indicates the need for their collective analysis to uncover valuable practices and potential research opportunities. To fulfill this need, we consolidate and unify the knowledge on SCE practices through a systematic literature review. The results show that there has been limited and unsystematic investigation of SCE by the community, highlighting the importance of creating and promoting guidelines for SCE practices. Therefore, we create SCENE, a comprehensive set of guidelines for systematically reporting SCE. The goal is to support the clarity, consistency, and reproducibility of SCE practices. SCENE guidelines are evaluated by cybersecurity practitioners and active researchers in the field, and is mapped to established methodological guidelines. The results indicates that SCENE is perceived positive in terms of usefulness, understandability, practicality, and completeness. SCENE is also found to complement established experimental reporting guidelines and bridge the gap between academic studies and industrial use.

Color fidelity is often overlooked in simulation-based validation for autonomous vehicles, yet even minor color mismatches can undermine the reliability of AI-driven perception systems. In this paper, we systematically examine how controlled deviations in color reproduction-quantified by$\Delta E$-affect object detection accuracy across 32 variants of YOLO. Using a Macbeth ColorChecker, we derive calibrations for key color transforms (brightness, contrast, hue, gamma, saturation and color bias) and apply these to the COCO validation set. Our evaluations demonstrate that increasing$\Delta E$yields significant drops in detection metrics, especially for safety-critical categories such as pedestrians and cyclists. Based on these findings, we propose$\Delta E$thresholds that define acceptable color fidelity in camera simulations (e.g.,$\Delta E\leq 3$for$\Delta \mathbf{mAP}\leq 1\%)$. Further-more, we contribute these transformed datasets and scripts as a publicly available benchmark, enabling reproducible comparisons and guiding future research on color-based vulnerabilities in automated driving and other safety-critical domains.

Large Language Models (LLMs) are increasingly used by software engineers for code generation. However, limitations of LLMs such as irrelevant or incorrect code have highlighted the need for prompt programming (or prompt engineering) where engineers apply specific prompt techniques (e.g., chain-of-thought or input-output examples) to improve the generated code. While some prompt techniques have been studied, the impact of different techniques—and their interactions— on code generation is still not fully understood. In this study, we introduce CodePromptEval, a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated by three LLMs (GPT-4o, Llama3, and Mistral). Our findings show that while certain prompt techniques significantly influence the generated code, combining multiple techniques does not necessarily improve the outcome. Additionally, we observed a trade-off between correctness and quality when using prompt techniques. Our dataset and replication package enable future research on improving LLM-generated code and evaluating new prompt techniques.

Security assurance cases (SACs) consist of arguments that are supported by evidence to justify that a system is acceptably secure. However, they are a relatively static representation of the system’s security and therefore currently not effective at runtime which make them difficult to maintain and unable to support users during threats. The aim of this paper is to investigate how SACs can be adapted to become more effective at runtime and increase confidence in the system’s security. We extend an example SAC with game theory, which models the interaction between the system and attacker and identifies their optimal strategies based on their payoffs and likelihoods. The extension was added as a security control in the assurance case, where a security claim indicates what strategy should be taken at runtime. This claim changes dynamically with the recommended strategy output by the game-theoretic model at runtime. Based on the results of the evaluation, the extension was considered to be potentially effective, however this would further depend on how it is implemented in practice.

Cybersecurity issues in medical devices threaten patient safety and can cause harm if exploited. Standards and regulations therefore require vendors of such devices to provide an assessment of the cybersecurity risks as well as a description of their mitigation. Security assurance cases (SACs) capture these elements as a structured argument. Compiling an SAC requires taking domain-specific regulations and requirements as well as the way of working into account. In this case study, we evaluate CASCADE, an approach for building SAC in the context of a large medical device manufacturer with an established agile development workflow. We investigate the regulatory context as well as the adaptations needed in the development process. Our results show the suitability of SACs in the medical device industry. We identified 17 use cases in which an SAC supports internal and external needs. The connection to safety assurance can be achieved by incorporating information from the risk assessment matrix into the SAC. Integration into the development process can be achieved by introducing a new role and rules for the design review and the release to production as well as additional criteria for the definition of done. We also show that SACs built with CASCADE fulfill the requirements of relevant standards in the medical domain such as ISO 14971.

With the increasing prevalence of open and connected products, cybersecurity has become a serious issue in safety-critical domains such as the automotive industry. As a result, regulatory bodies have become more stringent in their requirements for cybersecurity, necessitating security assurance for products developed in these domains. In response, companies have implemented new or modified processes to incorporate security into their product development lifecycle, resulting in a large amount of evidence being created to support claims about the achievement of a certain level of security. However, managing evidence is not a trivial task, particularly for complex products and systems. This paper presents a qualitative interview study conducted in six companies on the maturity of managing security evidence in safety-critical organizations. We find that the current maturity of managing security evidence is insufficient for the increasing requirements set by certification authorities and standardization bodies. Organisations currently fail to identify relevant artifacts as security evidence and manage this evidence on an organizational level. One part of the reason are educational gaps, the other a lack of processes. The impact of AI on the management of security evidence is still an open question.

Security Assurance Cases (SAC) are structured arguments and evidence bodies used to reason about the security of a certain system. SACs are gaining focus in the automotive industry, as the needs for security assurance are growing in this domain. However, the state-of-the-arts lack a mature approach able to suit the needs of the automotive industry. In this article, we present CASCADE, an asset-driven approach for creating SAC, which is inspired by the upcoming security standard ISO/SAE-21434 as well as the internal needs of automotive Original Equipment Manufacturers (OEMs). CASCADE also differentiates itself from the state-of-the-art by incorporating a way to reason about the quality of the constructed security assurance case. We created the approach by conducting an iterative design science research study. We illustrate the results using the example case of the road vehicle’s headlamp provided in the ISO standard. We also illustrate how our approach aligns well with the structure and content of the ISO/SAE-21434 standard, hence demonstrating the practical applicability of CASCADE in an industrial context.

Container-based virtualization is gaining popularity in different domains, as it supports continuous development and improves the efficiency and reliability of run-time environments. Different techniques are proposed for monitoring the security of containers. However, there are no guidelines supporting the selection of suitable techniques for the tasks at hand. We aim to support the selection and design of techniques for monitoring container-based virtualization environments. : First, we review the literature and identify techniques for monitoring containerized environments. Second, we classify these techniques according to a set of categories, such as technical characteristic, applicability, effectiveness, and evaluation. We further detail the pros and cons that are associated with each of the identified techniques. As a result, we present CONSERVE, a multi-dimensional decision support framework for an informed and optimal selection of a suitable set of container monitoring techniques to be implemented in different application domains. A mix of eighteen researchers and practitioners evaluated the ease of use, understandability, usefulness, efficiency, applicability, and completeness of the framework. The evaluation shows a high level of interest, and points out to potential benefits.

Traceability management relies on a supporting model, the traceability information model (TIM), that defines which types of relationships exist between which artifacts and contains additional constraints such as multiplicities. Constructing a TIM that is fit for purpose is crucial to ensure that a traceability strategy yields the desired benefits. However, which design decisions are critical in the construction of TIMs and which impact they have on the usefulness and applicability of traceability is still an open question. In this paper, we use two cases of TIMs constructed for safety-critical, automotive systems with industrial safety experts, to identify key design decisions. We also propose a comparison scheme for TIMs based on a systematic literature review and evaluate the two cases as well as TIMs from the literature according to the scheme. Based on our analyses, we thus derive key insights into TIM construction and the design decisions that ensure that a TIM is fit for purpose.

Abstract Security Assurance Cases (SAC) are a form of structured argumentation used to reason about the security properties of a system. After the successful adoption of assurance cases for safety, SAC are getting significant traction in recent years, especially in safety-critical industries (e.g., automotive), where there is an increasing pressure to be compliant with several security standards and regulations. Accordingly, research in the field of SAC has flourished in the past decade, with different approaches being investigated. In an effort to systematize this active field of research, we conducted a systematic literature review (SLR) of the existing academic studies on SAC. Our review resulted in an in-depth analysis and comparison of 51 papers. Our results indicate that, while there are numerous papers discussing the importance of SAC and their usage scenarios, the literature is still immature with respect to concrete support for practitioners on how to build and maintain a SAC. More importantly, even though some methodologies are available, their validation and tool support is still lacking.

Assurance cases are structured arguments that are commonly used to reason about the safety of a product or service. Currently, there is an ongoing push towards using assurance cases also for cybersecurity, especially in safety critical domains, like automotive. While the industry is faced with the challenge of defining a sound methodology to build security assurance cases, the state of the art is rather immature. Therefore, we have conducted a thorough investigation of the (external) constraints and (internal) needs that security assurance cases have to satisfy when used in the automotive industry. This has been done with 28 participants and in the context of two large automotive companies located in Europe: Company A is a passenger car manufacturer, while Company B is a truck manufacturer. An extended version of this paper is available online at https://arxiv.org/abs/2003.14106.