The field of Human-Centred Explainable AI (HCXAI) has been rapidly expanding. In turn, there has been an increase in the number of papers suggesting design principles for HCXAI. However, it is unclear the extent to which design requirements overlap between papers, and in turn what the field overall considers to be HCXAI design requirements. To overcome this, this study analysed the state of the field via a scoping review of papers suggesting HCXAI design requirements, and a Content Analysis of the extracted principles. A total of 330 design principles were identified from 35 papers, which were subsequently categorised into 43 codes and grouped into 4 main areas of focus. Based on these findings, we propose a definition of HCXAI which identifies HCXAI as a design process rather than an XAI technique. Finally, an overview of the current state of HCXAI is presented, as well as areas where further research is required.

Existing classification frameworks for AI and autonomous systems are being outpaced by recent advancements in AI technologies. This limits their applicability to modern intelligent systems, particularly agentic AI systems (autonomous systems that leverage foundation models to achieve wide-ranging, multi-layered goals). To address this deficiency, we introduce INSYTE, a multi-faceted framework that supports the classification of AI systems ranging from traditional rule-based systems to cutting-edge embodied AI and agentic systems. To that end, INSYTE considers the essential characteristics of an AI system across eight key dimensions grouped into four categories: system design ( underspecification and adaptiveness ); functionality ( breadth and depth ); operating environment ( diversity and dynamism ); and independence from human operational control ( intervention and oversight ). Different AI systems (or versions of systems) yield different ‘patterns’ on an eight-axis radar chart that INSYTE uses to provide an immediate visual summary of an AI system’s overall capability and a detailed representation of its individual characteristics. The INSYTE framework aligns with OECD’s definition of deployed AI systems, which is becoming the standard definition used by legislators and developers worldwide.

Autonomous systems are of high complexity and often regarded as mixed-criticality systems (MCSs) in which functions are allocated criticality levels according to risk assessment based on safety standards. Typically, tasks have different real-time requirements across criticality levels, and the estimated worst-case execution times (WCETs) are distinct. Further, limitations in computational resources increase the difficulty of integrating tasks onto one shared hardware platform. Conventionally, all nonsafety critical tasks must be discarded or suspended to guarantee the execution of safety-critical tasks when facing a timing fault. This typically leads to a considerable decrease in the system’s Quality-of-Service (QoS). Achieving more graceful degradation is critical to minimizing QoS reduction. This work focuses on tackling timing faults and proposes a novel graceful degradation strategy for use in a mixed-criticality context. Thus, when a system has multiple operational modes depending on the environment or an operational task, our approach can give an effective way of managing degradation to maximize QoS, which is currently not sufficiently recognized in MCS. Furthermore, the proposed causality analysis-based degradation process “bridges the gap” so functional dependencies are considered in scheduling design and thus leads to a graceful degradation that is both feasible and reasonable in functional and nonfunctional terms. The evaluations show that QoS can be better preserved using the proposed context-aware degradation process when compared with more conventional MCS scheduling approaches.

Time-sensitive networking (TSN) is being widely investigated to provide Ethernet capabilities for in-vehicle backbone communication. However, the gate control list (GCL), as a simple mechanism for achieving timing determinism for safety-critical traffic (ST) frames with hard deadlines, is too rigid to handle the intrinsic timing uncertainty of automated driving systems (ADSs). Due to the complexity and the unpredictable operating environment, there can be delayed ST frames that can disrupt the solidly fixed timing behavior. In this work, we present one novel approach to effectively use bandwidth resources to deal with delayed ST frames, which cannot be handled by a traditional fixed GCL, and the discarding of them should be reduced to improve the system’s integrity. An acceptance test is implemented to report to the application layer when an ST frame will miss its deadline and hence be rejected, i.e., prevented from entering the switch. To further improve the efficiency of bandwidth usage, we investigate how to improve the performance of the more important Class A frames in an audio-video-bridging (AVB) switch, which adopts a credit-based shaper mechanism, and we propose a constant bandwidth server to replace the credit-based shaper while taking fairness into consideration. Evaluation with extensive experiments shows that both resilience and efficiency of the TSN are significantly enhanced compared with a credit-based shaper, especially when the traffic load is relatively high. For the delayed ST frames and event-triggered traffic, our approach is able to schedule more than the solution using the AVB switch even with a high network load.

Software inspection is a static analysis technique that is widely used for defect detection, but which suffers from a lack of rigor. In this paper, we address this problem by taking advantage of formal specification and analysis to support a systematic and rigorous inspection method. The aim of the method is to use inspection to determine whether every functional scenario defined in the specification is implemented correctly by a set of program paths and whether every program path of the program contributes to the implementation of some functional scenario in the specification. The method is comprised of five steps: deriving functional scenarios from the specification, deriving paths from the program, linking scenarios to paths, analyzing paths against the corresponding scenarios, and producing an inspection report, and allows for a systematic and automatic generation of a checklist for inspection. We present an example to show how the method can be used, and describe an experiment to evaluate its performance by comparing it to perspective-based reading (PBR). The result shows that our method may be more effective in detecting function-related defects than PBR but slightly less effective in detecting implementation-related defects. We also describe a prototype tool to demonstrate the supportability of the method, and draw some conclusions about our work.

Generative programming and model transformation techniques are becoming widely used for the development of software components for product lines. The ability to develop components with identified common and variable parts, and rapidly instantiate product-specific versions is key to many software product line approaches. However if this approach is to be truly cost effective for high assurance applications, the instantiation process must be property-preserving, any verification evidence acquired on the product-line component must be demonstrably applicable to the instantiated component. In this paper we outline an approach that uses static analysis techniques and the SPARK language that can potentially demonstrate the correctness of model transformations.

Writing formal specifications can help developers understand users' requirements, and build a solid foundation for implementation. But like other activities in software development, it is error-prone, especially for large-scale systems. In practice, effective detection of specification errors still remains a challenge. In this paper, we put forward a rigorous, systematic method for the inspection of model-based formal specifications. The method makes good use of the well-defined consistency properties of a specification to provide precise rules and guidelines for inspection. The inspection process utilizes both well-defined expressions derived from the specification and human inspectors' judgments to find errors. We present a case study of the method by describing how it is applied to inspect an Automated Teller Machine (ATM) software specification to investigate the method's feasibility, and explore potential challenges in using it. We also describe a prototype software tool including its functions and distinct features to demonstrate the tool supportability of the method.

Experience in all aspects of software engineering has confirmed the pivotal role of focusing on architectural concerns in the development of complex software-intensive systems. Consequently, the past 20 years has seen significant investments in the theory and practice of software architecture. However, architectural deficiencies are frequently cited as a key factor in the shortcomings and failures that lead to unpredictable delivery of complex operational systems. Here, we consider the art and science of software architecture: we explore the current state of software architecture, identify key architectural trends, and directions in academia and industry, and highlight some of the architectural research challenges which need to be addressed. The paper proposes a detailed agenda of research activities to be carried out by a partnership between academia and industry. While challenges exist in many domains, for this paper we draw examples from one area of particular concern: safety-critical systems.

When developing a new domain-specific architectural style, there can be uncertainty about the feasibility of using that style. In particular, the HADES architectural style contains refactoring patterns intended to remove undesirable scheduling features such as deadlock and livelock, but these patterns have not yet been validated. We report on the development of a simulator environment to help validate these refactoring patterns and generally demonstrate HADES architectures to non-specialists. The simulator implements the synchronisation and coordination specified by the architecture to help visualise the behaviour of the otherwise static architectural descriptions. We found simulation to be a useful tool in both visualising complex interaction semantics and in validating refactoring patterns

Abstract Grid systems span multiple organizations, so their workflow processes have security requirements, such as restricting access to data or ensuring that process constraints are observed. These requirements are often managed by the workflow component, because of the close association between this sub‐system and the processes it enacts. However, high‐quality security mechanisms and complex functionality are difficult to combine, so designers and users of workflow systems are faced with a tradeoff between security and functionality, which is unlikely to provide confidence in the security implementation. This paper resolves that tension by showing that process security can be enforced outside the workflow component. Separating security and process functionality in this way improves the quality of security protection, because it is implemented by standard system mechanisms; it also allows the workflow component to be deployed as a standard service, rather than a privileged system component. To make this change of design philosophy accessible outside the security community it is documented as a collection of refactorings, which include problem templates that identify suspect design practice, and target patterns that provide solutions. Worked examples show that these patterns can be used in practice to implement practical applications, with both traditional workflow security concerns, and Grid requirements. Copyright © 2005 John Wiley & Sons, Ltd.

We show that landmark based localisation (LBL) and Lowe's model-based localisation (MBL) are complementary in that LBL provides a pose initialisation to MBL, which is a necessary input to the algorithm, and MBL can then refine that pose to give a more accurate pose estimate than LBL alone can provide. For LBL, we extend Betke and Gurvit's method, such that it can be used with standard perspective cameras (their original proposal was for omnidirectional cameras) in order to get a useful initial value as an input to Lowe's method. Intensive experiments have been carried out to analyse how camera parameters (intrinsic and extrinsic) affect the LBL position and orientation errors in the initial pose estimate. In error propagation experiments, we show that the position and orientation of a robot are sensitive to focal length and errors in imaged feature positions respectively. In the MBL pose refinement phase, we find that MBL is able to refine the position estimate, but the error in orientation estimate remains the same.

This paper investigates the implications of using modern superscalar processors in the safety-critical domain. Firstly, a description of current certification practice and devices is given as background. This is followed by an exposition of the certification argument for a processor when used in a safety-critical application. Throughout the presentation of the argument two types of modern processor are considered, commercial off-the-shelf (COTS) processors and purpose-designed bespoke devices. This allows the elaboration of positive and negative features of processors that can be used as part of the selection (for COTS) or design (for bespoke) process.

Abstract The mutation method assesses test quality by examining the ability of a test set to distinguish syntactic deviations representing specific types of faults from the program under test. This paper describes an empirical study performed to evaluate the effectiveness of object‐oriented (OO) test strategies using the mutation method. The test sets for the experimental system are generated according to three selected OO test strategies and their effectiveness is compared by determining how well the developed test sets kill injected mutants derived from an established mutation system Mothra and the authors' own OO‐specific mutation technique which is termed Class Mutation. Copyright © 2001 John Wiley & Sons, Ltd.

Complexity arises from many sources: both within and outside the system. Internal sources include modern hardware, e.g. super-scalar processors, and external sources include the requirements for evolving already successful systems. Complexity is inescapable unless we are willing to reduce our dependence on computers, and to forgo the benefits they bring us. This raises the issue of how we control, or simply cope with, ever-increasing complexity. We try to clarify what is meant by complexity, and what causes complexity. We identify some causes of complexity, focusing on embedded systems with strict dependability requirements, as these pose some of the most significant challenges. We then propose some strategies for coping with complexity, including the use of product families and the use of risk as a means of managing complexity.

This paper presents a technique for automatically generating test-data to test exceptions. The approach is based on the application of a dynamic global optimization based search for the required test-data. The authors' work has focused on test-data generation for safety-critical systems. Such systems must be free from anomalous and uncontrolled behaviour. Typically, it is easier to prove the absence of any exceptions than proving that the exception handling is safe. A process for integrating automated testing with exception freeness proofs is presented as a way forward for tackling the special needs of safety critical systems. The results of a number of simple case-studies are presented and show the technique to be effective. The major result shows the application of the technique to a commercial aircraft engine controller system as part of a proof of exception freeness. This illustrates how automated testing can be effectively integrated into a formal safety-critical process to reduce costs and add value. Copyright © 2000 John Wiley & Sons, Ltd.

We discuss issues in the development of formal methods for use in aerospace applications, reflecting our experience in working with both Rolls-Royce and British Aerospace. We discuss some of the key factors which we believe govern the application of discrete mathematics to aerospace applications, drawing comparisons with applied engineering mathematics in other domains. We give an overview of three projects (the three "small steps"): the development of a domain-specific language for aircraft engine control system specification; the development of a formal semantics and tool support for state transition systems to facilitate analysis of specifications produced by systems engineers; the use of formalism in support of test automation. We then discuss the "gap" we see between the needs of industry and the current focus of the formal methods research community by pointing out important facets of industrial applicable formal methods which are not receiving adequate attention. We refer to this as a "giant leap" due to the need for a cultural shift in the research community and the need for a coherent approach to the identified research issues rather than piecemeal studies of the issues. Our conclusions are to be optimistic for the future use of formal methods in industry albeit with concern that their potential will not be realised unless there is a shift in emphasis within the research community?.

Structural testing criteria are mandated in many software development standards and guidelines. The process of generating test data to achieve 100% coverage of a given structural coverage metric is labour-intensive and expensive. This paper presents an approach to automate the generation of such test data. The test-data generation is based on the application of a dynamic optimisation-based search for the required test data. The same approach can be generalised to solve other test-data generation problems. Three such applications are discussed-boundary value analysis, assertion/run-time exception testing, and component re-use testing. A prototype tool-set has been developed to facilitate the automatic generation of test data for these structural testing problems. The results of preliminary experiments using this technique and the prototype tool-set are presented and show the efficiency and effectiveness of this approach.

Reusability is widely suggested to be a key to improving software development productivity and quality. It has been further argued that reuse at the requirements level can significantly increase reuse at the later stages of development. However, there is little evidence in the literature to suggest that requirements reuse is widely practised. This paper describes ten practical steps towards systematic requirements reuse based on work at the Rolls-Royce University Technology Centre (UTC) for Rolls-Smiths Engine Controls Ltd. (RoSEC) in the domain of aero-engine control systems. We believe these steps have made a significant overall contribution to the 50% reuse figure quoted by the management at RoSEC for current projects within the BR700 family of engine controllers.

The safety argument manager, SAM, is a tool to support the process of developing safety cases (Fodder, J. et al., see Proc. Safety-Critical Systems Symposium, Bristol, 1993). In SAM a safety case is expressed by a goal structure and associated solutions. Each solution is expressed in Toulmin argument form (1984). Fault trees can be constructed and attached to a goal. We have constructed the tool SAM, and investigated how these three different notations can be used in developing a real safety case for a complex system by completing a safety case study. This paper presents the case study, using SAM, of the PES (Programmable Electronic Systems) example (PES-programmable electronic systems in safety related applications, Health and Safety Executive, p.80-99, 1987).

Procurers of critical computer based systems have to assess the suitability of implementations provided by external contractors. What an assessor requires is a clear, comprehensible and defensible argument, with supporting evidence, that a system will behave acceptably. We describe how the Goal Structuring Notation (GSN) can be used to capture suitability arguments with supporting evidence attached in the form of design models, test results, analysis results, audit reports, etc. We also describe associated tool support-the Safety Argument Manager (SAM). We describe work being carried out by the Defence Research Agency (DRA) and the University of York supported by the UK Ministry of Defence's (MoD) Strategic Research Programme. It presents the preliminary results and expected future direction of the project. Nothing in this paper should be taken as the official position of the MoD or the DRA.

Safety Critical Systems are those systems that can potentially lead to loss of life, injury, and environmental damage. Therefore such systems have to be designed and built to meet a variety of functional and non-functional requirements, including safety, reliability, availability, and maintainability. It is essential to assess, as an independent activity, the extent to which these requirements have been met, and for complex systems there is no single analysis technique which can be employed. It is therefore necessary to use a number of different safety (and reliability) analysis techniques to perform an assessment. Using a variety of techniques raises issues of consistency—if the individual analyses and models are inconsistent with respect to each other then the overall assessment is likely to be inconsistent, and therefore not trustworthy. In this paper we present a set of rules that should hold between a representative set of safety analysis techniques, demonstrate how they can be enforced and checked by an underpinning data model, and describe a software tool (based on these ideas) to support integrated safety analysis.

Abstract CADi began as a type‐checker for Z specifications, which were embedded within troff documents. In adding other functionality, such as expansion of schema calculus expressions, and alternative functionality, such as an embedding within LATEX documents, a more open architecture was created into which new tools can be added and alternative tools substituted. This paper gives an overview of CADiZ and describes its open architecture. Many of the ideas used in implementing CADiZ would be applicable to other language processing systems.

A composable security property of noninterference is applied to carry out a case study of the security analysis of an information system comprising a number of components. We first adopt a different notation to generalise this property for the enforcement of different security requirements. This modified property is then applied to assess the security of the information system. Finally, useful observations derived from this case study are discussed, which can help to develop cost-effective approaches for the design and evaluation of secure systems.>

As we know, current hookup or composable properties may impose over-strong security requirements on component systems. To overcome this problem, connectivities of the components have to be considered in order to appropriately handle their composition. Based on such a consideration, in this paper we adopt the concept of rely- and guarantee-conditions to present a composable property of noninterference. We enforce the requirement of noninterference only on some input-output entities of each component with regard to its connectivity, and communication constraints on its others so as to ensure that their entire system can satisfy noninterference. This enables the system and its components to possess different security properties, i.e. the security property of the system can be logically stronger than security properties of its components.>

Current hookup theories may impose overstrong security requirements on component systems. To overcome this problem, connectivities of the components may have to be considered in order to appropriately handle their composition. Such a consideration is used here to describe composable security properties. Security requirements are enforced only on some input and output entities of each component with regard to its connectivity, and communication constraints on its others so as to ensure that their entire system can satisfy its security requirement. This enables the system and its components to possess different security properties, i.e., the security property of the system can be logically stronger than security properties of its components.>

Ideally, a measure of the security of a system should capture quantitatively the intuitive notion of ‘the ability of the system to resist attack’. That is, it should be operational, reflecting the degree to which the system can be expected to remain

Composability properties of component systems are addressed. By means of analysis of external relations among components, security problems associated with composition of the components are investigated. To solve these problems, two security models are presented. By comparing these two models, important properties of secure composition of component systems are identified.>

We are now starting to see the first applications of formal methods to the development of safety-critical computer based systems. Discussion on what are appropriate methods and tools is still intense, and there is no standard approach that presents a complete solution for the formal development of such systems. Some of the protagonists claim that formal methods offer a complete solution to the problems of safety-critical software development. Others claim that formal methods are of little or no use – or at least that their utility is severely limited by the cost of applying the techniques. The aim of this paper is to try to cast some light on this debate and to discuss from a technico-philosophical viewpoint the benefits and limitations of formal methods in this context.

We consider the distinction between the terms 'safety' and 'security' in terms of the differences in causal structure and in terms of the differences in the degree of harm caused. The discussion is illustrated by an analysis of a number of cases of system failure where the safety and security issues seem, at least at first sight, to be difficult to disentangle.

The paper presents a formal and systematic model for analysis and testing of secure systems. The concept of security dependency is first introduced, and certain rules and theorems of security dependency are then formally described. These rules can be used as a basis for static analysis, dynamic testing, and covert channel analysis for a secure system. The major feature of the model presented is that static analysis and dynamic testing can be combined together to evaluate the security properties of a system.>

The aim is to discuss the nature of safety arguments to consider the role of system and software reliability evaluation in these arguments, and to outline an approach to supporting the development of safety arguments. The author reviews some existing work addressing the problems of evaluating systems to high levels of reliability such as 10/sup -9/ failures per hour using 'black box' testing. He also considers ways of achieving confidence beyond testable levels through the use of prior beliefs and discusses some approaches to achieving strong prior beliefs. He uses these possible approaches to illustrate a canonical form for representing (safety) arguments, and to outline the characteristics of a tool which he is constructing for safety argument management.>

The authors first describe some issues that arise from the interplay between the security requirements for an integrated project support environment (IPSE) for the development of a trusted system, and the security requirements of the trusted system itself. All of these issues derive from security policy and the modeling of security policy. A framework is then presented which allows security policies to be expressed in the context of the enterprise whose needs the trusted system is intended to serve. Finally some possible applications of the framework are used to indicate how security policies affect design decision-making, security policy conflict detection, and security risk evaluation.>