CONTEXT: Assurance Cases (ACs) are prepared to argue that the system’s desired quality attributes (e.g., safety or security) are satisfied. While there is strong adoption of ACs, practitioners are often left asking an important question: are we confident that the claims made by the case are true? While many confidence assessment methods (CAMs) exist, little is known about the use of these methods in practice. OBJECTIVE: Develop an understanding of the current state of practice for AC confidence assessment: what methods are used in practice and what barriers exist for their use? METHOD: Structured interviews and an email questionnaire were used to gather data from practitioners with experience contributing to real-world ACs. Open-coding was performed on transcripts. A description of the current state of AC practice and future considerations for researchers was synthesized from the results. RESULTS: A total of n = 19 practitioners were interviewed. The most common CAMs were (peer-)review of ACs, dialectic reasoning (“defeaters”), and comparing against checklists. Some practitioners also used models to gain confidence in an AC. Participants preferred qualitative methods and expressed concerns about quantitative CAMs. Barriers to using CAMs included additional work, inadequate guidance, subjectivity and interpretation of results, and trustworthiness of methods. CONCLUSION: While many CAMs are described in the literature there is a gap between the proposed methods and needs of practitioners. Researchers working in this area should consider the need to: connect CAMs to established practices, use CAMs to communicate with interest holders, crystallize the details of CAM application, curate accessible guidance, and confirm that methods are trustworthy.

Assurance cases (AC) are structured arguments that justify why a system is acceptably safe. Though ACs can increase confidence that systems will operate safely and reliably, they are also susceptible to problems such as reasoning errors and confirmation bias. Recent work proposed AI-Supported Eliminative Argumentation (AI-EA), a framework leveraging Generative AI (GAI) models to support AC development by identifying potential reasons why the argument may be invalid (a.k.a. defeaters) so that they can be mitigated. However, this framework was not implemented and its effectiveness was not assessed empirically.In this practical experience paper, we implement AI-EA, explain and justify our design choices, and report on our practical experience in empirically evaluating its effectiveness in collaboration with experts in the safety domain. Our evaluation considers 171 AI-generated defeaters across two industrial case studies from the nuclear and automotive domains. Our findings show that GAI can generate informative defeaters with few significant hallucinations and that 25% of the generated defeaters were confirmed by developers of each AC to represent reasonable doubts or errors in the argument. Our implementation and data are made publicly available.

Self-adaptive systems are able to change their behaviour at run-time in response to changes. Self-adaptation is an important strategy for managing uncertainty that is present during the design of modern systems, such as autonomous vehicles. However, assuring the safety of self-adaptive systems remains a challenge, particularly when the adaptations have an impact on safety-critical functions. The field of safety engineering has established practices for analyzing the safety of systems. System Theoretic Process and Analysis (STPA) is a hazard analysis method that is well-suited for self-adaptive systems. This paper describes a design-time extension of STPA for self-adaptive systems. Then, it derives a reference model and analysis obligations to support the STPA activities. The method is applied to three self-adaptive systems described in the literature. The results demonstrate that STPA, when used in the manner described, is an applicable hazard analysis method for safety-critical self-adaptive systems.

In the recent decades computing systems have become ubiquitous in our daily life. Due to wear and tear, limited component lifetime, and extraneous factors, among other reasons, all of the systems that we design and implement are subject to failure. One of the main areas in the field of fault tolerance, system evaluation, is concerned with the analysis of systems and faults as well as their operational environments. In the context of system evaluation, this paper is concerned with failure modeling and fault prediction. We propose a model for evaluating network systems in the context of failure and repair. Although the focus here is on sensor networks, it can surely be extended to other situations. A systems engineer can use the proposed model to estimate the longevity of a system and plan appropriate maintenance during the system design or maintenance phases. The approach makes use of Markov chains to model failure states of the system based on historical data. The effectiveness of this model is demonstrated through preliminary experiments and a case study, which also confirm intuitions about the effects of network topology on the network's reliability.