Constructing assurance cases is a widely used and sometimes required process toward demonstrating that safety-critical systems will operate safely in their planned environment. To mitigate the risk of errors and missing edge cases, the concept of defeaters - challenges to claims in an assurance case - has been introduced. Defeaters can detect weaknesses in the arguments, prompting further investigation and timely mitigations. However, capturing defeaters relies on expert judgment, experience, and creativity and must be done iteratively due to evolving requirements and regulations. In this paper, we propose CoDefeater, an automated process to leverage large language models (LLMs) for finding defeaters. Initial results on two systems show that LLMs can efficiently find known and unforeseen feasible defeaters to support safety analysts in enhancing the completeness and confidence of assurance cases.

Variability constraints are an integral part of the requirements for a configurable system. The constraints specified in the requirements on the legal combinations of options define the space of potential valid configurations for the system-to-be. This paper reports on our experience with the variability-related requirements constraints of a flight software framework used by multiple space missions. A challenge that we saw for practitioners using the current framework, now open-sourced, is that the specifications of its variability-related requirements and constraints are dispersed across several documents, rather than being centralized in the software requirements specification. Such dispersion can contribute to misunderstandings of the side-effects of design choices, increased effort for developers, and bugs during operations. Based on our experience, we propose a new software variability model, similar to a product-line feature model, in the flight software framework. We describe the structured technique by which our model is developed, demonstrate its use, and evaluate it on a key service module of the flight software. Results show that our lightweight modeling technique helped find missing and inconsistent variability-related requirements and constraints. More generally, we suggest that a variability modeling technique such as this can be an efficient way for developers to centralize the specification and improve the analysis of dispersed variability-related requirements and constraints in other configurable systems.

SafeWalk is a tool kit designed for simulation of a safety-critical product line of astronaut jetpacks. It provides (1) a Unity-based simulation development environment and (2) software artifacts inspired by a real jetpack used by astronauts during spacewalks. SafeWalk has been developed to be a readily extensible and real-time configurable product line for use in research and education. It provides a rich environment conducive to empirical research into safety-critical requirements and to visualization of safety-related human-cyberphysical interactions.

The field of cyber-molecular systems is growing rapidly. In these nanotechnology applications the computational logic is encoded by developers into the molecules themselves. Many planned applications are safety-critical, including bio-compatible sensors, pollution trackers, and targeted drug-delivery devices. Requirements engineering (RE) activities and artifacts are essential to assuring the safety of molecular programs. However, molecular programmed devices offer challenges to traditional RE activities. Molecular programmed systems are nanoscale, so hard to monitor; execute at scale, typically 1010devices in solution at once; and have probabilistic behavior. Toward safe molecular programs, we propose a new framework, RE4DNA, for their safety requirements discovery, specification, and verification. Its contribution is to bridge the cyber and the molecular in the requirements engineering process. Further, use of RE4DNA identifies building blocks that can contribute to a preliminary safety case. In this paper we introduce RE4DNA, describe how it handles some particular challenges of molecular programming, illustrate its use on a benchmark molecular program, and discuss future work.

The growing adoption of unmanned aerial vehicles (UAVs) for tasks such as eCommerce, aerial surveillance, and environmental monitoring introduces the need for new safety mechanisms in an increasingly cluttered airspace. In our work we thus emphasize safety issues that emerge at the intersection of infrastructures responsible for controlling the airspace, and the diverse UAVs operating in their space. We build on safety assurance cases (SAC) - a state-of-the-art solution for reasoning about safety - and propose a novel approach based on interlocking SACs. The infrastructure safety case (ISAC) specifies assumptions upon UAV behavior, while each UAV demonstrates compliance to the ISAC by presenting its own (pluggable) safety case (pSAC) which connects to the ISAC through a set of interlock points. To collect information on each UAV we enforce a “trust but monitor” policy, supported by runtime monitoring and an underlying reputation model. We evaluate our approach in three ways: first by developing ISACs for two UAV infrastructures, second by running simulations to evaluate end-to-end effectiveness, and finally via an outdoor field-study with physical UAVs. The results show that interlocking SACs can be effective for identifying, specifying, and monitoring safety-related constraints upon UAVs flying in a controlled airspace.

We show that very simple molecular systems, modeled as chemical reaction networks, can have behaviors that exhibit dramatic phase transitions at certain population thresholds. Moreover, the magnitudes of these thresholds can thwart attempts to use simulation, model checking, or approximation by differential equations to formally verify the behaviors of such systems at realistic populations. We show how formal theorem provers can successfully verify some such systems at populations where other verification methods fail.

Safety Assurance Cases (SACs) are increasingly used to guide and evaluate the safety of software-intensive systems. They are used to construct a hierarchically organized set of claims, arguments, and evidence in order to provide a structured argument that a system is safe for use. However, as the system evolves and grows in size, a SAC can be difficult to maintain. In this paper we utilize design science to develop a novel solution for identifying areas of a SAC that are affected by changes to the system. Moreover, we generate actionable recommendations for updating the SAC, including its underlying artifacts and trace links, in order to evolve an existing safety case for use in a new version of the system. Our approach, Safety Artifact Forest Analysis (SAFA), leverages traceability to automatically compare software artifacts from a previously approved or certified version with a new version of the system. We identify, visualize, and explain changes in a Delta Tree. We evaluate our approach using the Dronology system for monitoring and coordinating the actions of cooperating, small Unmanned Aerial Vehicles. Results from a user study show that SAFA helped users to identify changes that potentially impacted system safety and provided information that could be used to help maintain and evolve a SAC.

Augmented reality (AR) systems excel at offering users real-time, situation-aware information to support users' decision making. With AR, rich visualizations of relevant data can be displayed to users without blocking their view of the real world. For example, an AR-enabled automotive windshield can display a red outline around a pedestrian to alert a driver starting a turn into that cross street. Other critical uses of AR applications that are or will soon be deployed include surgery, emergency response, vehicle maintenance, and pilot training. Many of these applications can enhance operational safety. However, developing risk analysis methods to handle failure modes in the melded virtual and physical realities remains an open problem. This paper proposes a risk analysis method with which to study computer-generated AR visualizations of system and environment states. The analysis framework incorporates three levels at which AR interfaces with the user: perception, comprehension, and decision-making. This method enables broader risk analysis of the entire cyber-physical-human system that an AR application may indirectly control. Preliminary results show that this method yields improved coverage of user-involved failure modes over current approaches. While the focus here is on safety, the method also appears applicable to AR security risks.

Molecular programming uses the computational power of DNA and other biomolecules to create useful nanoscale systems. Molecular program applications being developed include medical sensors that can be absorbed by the body after use, drug capsules that open only when they find diseased cells, and programmable nanoscale robots. This tutorial introduces the model-based language commonly used to write the requirements for molecular programs. This high-level modeling language is mathematically simple, very general, and well documented. Importantly, specifications written in it can be automatically compiled into implementable, detailed design descriptions. Participants will leave knowing how to write the requirements for some small molecular system components, where to go to learn more, and what are some open problems for writing the requirements of large molecular programs.

Problems with the correctness and completeness of environmental assumptions contribute to many accidents in safety-critical systems. The problem is exacerbated when products are modified in new releases or in new products of a product line. In such cases existing sets of environmental assumptions are often carried forward without sufficiently rigorous analysis. This paper describes a new technique that exploits the traceability required by many certifying bodies to reason about the likelihood that environmental assumptions are omitted or incorrectly retained in new products. An analysis of over 150 examples of environmental assumptions in historical systems informs the approach. In an evaluation on three safety-related product lines the approach caught all but one of the assumption-related problems. It also provided clearly defined steps for mitigating the identified issues. The contribution of the work is to arm the safety analyst with useful information for assessing the validity of environmental assumptions for a new product.

Developers maintaining safety-critical systems need to assess the impact a proposed change would have upon existing safety controls. By leveraging the network of traceability links that are present in most safety-critical systems, we can push timely information about related hazards, environmental assumptions, and safety requirements to developers. In this work we take a design science approach to discover the informational needs of developers as they engage in software maintenance activities and then propose and evaluate techniques for presenting and visualizing this information. Through a human-centered study involving five safety-critical system practitioners and 14 experienced developers, we analyze the way in which developers use requirements knowledge while maintaining safety-critical code, identify their informational needs, and propose and evaluate a supporting visualization technique. The insights proposed as a result of this study can be used to design requirements-based knowledge tools for supporting developers' maintenance tasks.

This talk looks backward at how the requirements engineering field and mission have changed in the 25 years since RE'17.

Molecular programming of DNA makes use of computer-aided design, a programming language to encode the design, and a a compiler that compiles the code into a list of corresponding DNA strands. To implement the code, the developer orders the DNA from a supplier and then runs the molecular program in the laboratory. Instruments such as spectrofluorometers and atomic force microscopes help verify that the molecular program ran correctly, self-assembling to achieve the intended structure and behavior. Applications of molecular programming that are on the horizon for commercial production include targeted and personalized medical applications, biosensors for air and water, molecular robots that can navigate in cells, and biocompatible computational devices. In this talk I describe the experiences and results of our interdisciplinary team in applying goal-oriented requirements techniques, reaction network modeling, and probabilistic model checking to molecular programmed nanosystems. Requirements challenges include handling scalability to very large numbers of devices or services (roughly similar to the internet of things), dealing with probabilistic behavior, and understanding the limits of safe operation in changing (literally fluid) environments. I discuss how computational tools and thinking assist the development efforts and what limitations currently hamper our requirements analysis in practice. More broadly, I suggest that the approaches useful for molecular programming may help with understanding the requirements for other large, distributed, autonomous systems.

Dynamic systems in DNA nanotechnology are often programmed using a chemical reaction network (CRN) model as an intermediate level of abstraction. In this paper, we design and analyze a CRN model of a watchdog timer, a device commonly used to monitor the health of a safety critical system. Our process uses incremental design practices with goal-oriented requirements engineering, software verification tools, and custom software to help automate the software engineering process. The watchdog timer is comprised of three components: an absence detector, a threshold filter, and a signal amplifier. These components are separately designed and verified, and only then composed to create the molecular watchdog timer. During the requirements-design iterations, simulation, model checking, and analysis are used to verify the system. Using this methodology several incomplete requirements and design flaws were found, and the final verified model helped determine specific parameters for biological experiments.

We propose an extension of van Lamsweerde's goal-oriented requirements engineering to the domain of programmable DNA nanotechnology. This is a domain in which individual devices (agents) are at most a few dozen nanometers in diameter. These devices are programmed to assemble themselves from molecular components and perform their assigned tasks. The devices carry out their tasks in the probabilistic world of chemical kinetics, so they are individually error-prone. However, the number of devices deployed is roughly on the order of a nanomole (a 6 followed by fourteen 0s), and some goals are achieved when enough of these agents achieve their assigned subgoals. We show that it is useful in this setting to augment the AND/OR goal diagrams to allow goal refinements that are mediated by threshold functions, rather than ANDs or ORs. We illustrate this method by engineering requirements for a system of molecular detectors (DNA origami “pliers” that capture target molecules) invented by Kuzuya, Sakai, Yamazaki, Xu, and Komiyama (2011). We model this system in the Prism probabilistic symbolic model checker, and we use Prism to verify that requirements are satisfied, provided that the ratio of target molecules to detectors is neither too high nor too low. This gives prima facie evidence that software engineering methods can be used to make DNA nanotechnology more productive, predictable and safe.

There is a lack of published studies providing empirical support for the assumption at the heart of product line development, namely, that through structured reuse later products will be less fault-prone. This paper presents results from an empirical study of pre-release fault and change proneness from four products in an industrial software product line. The objectives of the study are (1) to determine the association between various software metrics, as well as their correlation with the number of faults at the component level, (2) to characterize the fault and change proneness at various degrees of reuse, and (3) to determine how existing products in the software product line affect the quality of subsequently developed products and our ability to make predictions. The research results confirm, in a software product line setting, the findings of others that faults are more highly correlated to change metrics than to static code metrics. Further, the results show that variation components unique to individual products have the highest fault density and are the most prone to change. The longitudinal aspect of our research indicates that new products in this software product line benefit from the development and testing of previous products. For this case study, the number of faults in variation components of new products is predicted accurately using a linear model built on data from the previous products.

DNA nanotechnology uses the information processing capabilities of nucleic acids to design self-assembling, programmable structures and devices at the nanoscale. Devices developed to date have been programmed to implement logic circuits and neural networks, capture or release specific molecules, and traverse molecular tracks and mazes. Here we investigate the use of requirements engineering methods to make DNA nanotechnology more productive, predictable, and safe. We use goal-oriented requirements modeling to identify, specify, and analyze a product family of DNA nanodevices, and we use PRISM model checking to verify both common properties across the family and properties that are specific to individual products. Challenges to doing requirements engineering in this domain include the error-prone nature of nanodevices carrying out their tasks in the probabilistic world of chemical kinetics, the fact that roughly a nanomole (a 1 followed by 14 0s) of devices are typically deployed at once, and the difficulty of specifying and achieving modularity in a realm where devices have many opportunities to interfere with each other. Nevertheless, our results show that requirements engineering is useful in DNA nanotechnology and that leveraging the similarities among nanodevices in the product family improves the modeling and analysis by supporting reuse.

In previous work we described Diagnostic Tree for Verification (DTV), a partially automated software engineering technique by which diagnostic trees generated from system models are used to help check out diagnostic procedures. Diagnostic procedures are instructions used to isolate failures during operations. Assuring such procedures manually is time-consuming and costly. This paper reports our recent experience in applying DTV to diagnostic procedures for lighting failures in NASA's Habitat Demonstration Unit (HDU), a prototype for astronauts' living quarters. DTV identified missing and inconsistent instructions, as well as more-efficient sequences of diagnostic steps. Unexpectedly, the most significant benefit was finding assumptions that will not remain true as the system evolves. We describe both the challenges faced in applying DTV and how its independent perspective helped in assuring the procedures' adequacy and quality. Finally, the paper discusses more generally how software systems that are model-based, rapidly evolving and safety-critical appear most likely to benefit from this approach.

Reliability is important to software product-line developers since many product lines require reliable operation. It is typically assumed that as a software product line matures, its reliability improves. Since post-deployment failures impact reliability, we study this claim on an open-source software product line, Eclipse. We investigate the failure trend of common components (reused across all products), highreuse variation components (reused in five or six products) and low-reuse variation components (reused in one or two products) as Eclipse evolves. We also study how much the common and variation components change over time both in terms of addition of new files and modification of existing files. Quantitative results from mining and analysis of the Eclipse bug and release repositories show that as the product line evolves, fewer serious failures occur in components implementing commonality, and that these components also exhibit less change over time. These results were roughly as expected. However, contrary to expectation, components implementing variations, even when reused in five or more products, continue to evolve fairly rapidly. Perhaps as a result, the number of severe failures in variation components shows no uniform pattern of decrease over time. The paper describes and discusses this and related results. Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics—Product metrics,

Agent-oriented software engineering (AOSE) has provided powerful and natural, high-level abstractions in which software developers can understand, model and develop complex, distributed systems. Yet, the realization of AOSE partially depends on whether agent-based software systems can achieve reductions in development time and cost similar to other reuse-conscious development methods. Specifically, AOSE does not adequately address requirements specifications as reusable assets. Software product line engineering is a reuse technology that supports the systematic development of a set of similar software systems through understanding, controlling, and managing their common, core characteristics and their differing variation points. In this article, we present an extension to the Gaia AOSE methodology, named Gaia-PL (Gaia-Product Line), for agent-based distributed software systems that enables requirements specifications to be easily reused. We show how our methodology uses a product line perspective to promote reuse in agent-based software systems early in the development life cycle so that software assets can be reused throughout system development and evolution. We also present results from an application to show how Gaia-PL provided reuse that reduced the design and development effort for a large, multiagent system.

With the increasing reliance of complex real-world applications on composite web services assembled from independently developed component services, there is a growing need for effective approaches to verifying that a composite service not only offers the required functionality but also satisfies the desired non-functional requirements (NFRs). In high-assurance applications such as traffic control, medical decision support, and coordinated response to civil emergencies, of special concern are NFRs having to do with security, safety and reliability of composite services. Current approaches to verifying NFRs of composite services (as opposed to individual services) remain largely ad-hoc and informal in nature. In this paper we develop techniques for ensuring that a composite service meets the user-specified NFRs expressible in the form of hard constraints e.g., “response time has to be less than 5 minutes.” We introduce an automata-based framework for verifying that a composite service satisfies the desired NFRs based on the known guarantees regarding the non-functional properties of the component services. We further show how to improve the efficiency of verifying that a composite service indeed satisfies a desired set of NFRs by: (i) Exploiting information about the applicability of specific NFRs (e.g., security) only to certain subsets of the component services that make up a composite service to minimize the verification effort and (ii) Identifying inconsistencies between NFRs with overlapping scopes. We illustrate how our approach can be used to verify the security requirements for an Emergency Management System. We also show how the approach can be used to verify whether a composite service satisfies any desired set of NFRs that can be expressed in the form of hard constraints of a quantitative nature.

A principal goal of agent-oriented software engineering (AOSE) is to provide the mechanisms for reusing, maintaining and allowing the evolution of agent-based software systems. Our AOSE methodology, Gaia-PL, enables the design and development of multi-agent system product lines (MAS-PL)1by providing the software engineering processes to define and reuse requirements specifications and design artifacts. In this paper we extend our Gaia-PL methodology with automated tool support to enable the reuse and verification of MAS-PL requirements to better facilitate specification reuse during both initial system development and evolution. Specifically, we show how use of our product-line requirements management and verification tool along with feature modeling can support correct variation point selection, reuse and MAS-PL evolution. We illustrate and evaluate this work through an application to a proposed NASA agent-based pico-spacecraft swarm.

The contribution of this paper is a technique to assess whether software failures during operational use can invalidate the trust assumptions and, hence, the adequacy of the software security. Use of the technique identified four security-related software requirements for making the system more robust to denial-of-service attacks.

For many safety-critical systems a safety case is built as part of the certification or acceptance process. The safety case assembles evidence to justify that the design and implementation of a system avoid hazardous software behavior. Fault modeling and analysis can provide a rich source of evidence that the design meets safety goals. However, there is currently little guidance available to bridge the gap between the fault modeling that developers perform and the mandated safety case. In this experience report we describe results and open issues from an investigation of how evidence from software tool supported fault modeling and analysis of a spacecraft power system could assist in safety-case construction. The ways in which the software fault models can provide evidence for the safety case appears to be applicable to other critical systems.

NASA is, with the rest of industry, turning to product-line engineering to reduce costs and improve quality by effectively managing reuse. Experience in industry has shown that it is the verifiable conformance of each system to the product-line specifications that makes or breaks the product-line practice. Verification that the software for each project satisfies its intended product-line constraints is thus essential. This paper reports early results from aneffort to assemble from previous, industrial experience a set of enablers of verifiable conformance for use in the application engineering of NASA product lines. Lessons learned may be useful for developers of safety-critical, long-lived, or highly autonomous productlines, as well as for companies that integrate product line subsystems developed by multiple contractors.

Run-time monitoring is a powerful approach for dynamically detecting faults or malicious activity of software systems. However, there are often two obstacles to the implementation of this approach in practice: (1) that developing correct and/or faulty behavioral patterns can be a difficult, labor-intensive process, and (2) that use of such pattern-monitoring must provide rapid turn-around or response time. We present a novel data structure, called extended action graph, and associated algorithms to overcome these drawbacks. At its core, our technique relies on effectively identifying and caching specifications from (correct/faulty) patterns learned via machine-learning algorithm. We describe the design and implementation of our technique and show its practical applicability in the domain of security monitoring of sendmail software

We propose a specification-driven approach to Web service composition. The proposed framework allows users to start with a high-level, possibly incomplete specification of a desired (goal) service that is to be realized using a subset of the available component services. These services are represented by the system using transition systems augmented with guards over variables with infinite domains and are used to determine a strategy for their composition that would realize the goal service. In the event that the goal service cannot be realized using the available services, the system identifies the cause(s) for such failure which can then be used by the developer to reformulate the goal specification. Thus, the system supports Web service composition through iterative refinement of the functional specifications. We present a prototype implementation in tabled-logic programming environment that illustrates the key features of the proposed approach

Advances in software autonomy can support system robustness to a broader range of operational anomalies, called contingencies, than ever before. Contingency management includes, but goes beyond, traditional fault protection. Increased autonomy to achieve contingency management brings with it the challenge of how to verify that the software can detect and diagnose contingencies when they occur. The approach used in this work to investigate the verification was two-fold: (1) to integrate in a single model the representation of the contingencies and of the data signals and software monitors required to identify those contingencies, and (2) to use tool-supported verification of the diagnostics design to identify gaps in coverage of the contingencies. Results presented here indicate that tool-supported verification of the adequacy and correct behavior of such diagnostic software for contingency management can improve on-going contingency analysis, thereby reducing the risk that change has introduced gaps in the contingency software

This paper describes the use of obstacle analysis to identify anomaly-handling requirements for a safety-critical, autonomous system. The software requirements for the system evolved during operations due to an on-going effort to increase the autonomous system's robustness. The resulting increase in autonomy also increased system complexity. This investigation used obstacle analysis to identify and to reason incrementally about new requirements for handling failures and other anomalous events. Results reported in the paper show that obstacle analysis complemented standard safety-analysis techniques in identifying undesirable behaviors and ways to resolve them. The step-by-step use of obstacle analysis identified potential side effects and missing monitoring and control requirements. Adding an availability indicator and feature-interaction patterns proved useful for the analysis of obstacle resolutions. The paper discusses the consequences of these results in terms of the adoption of obstacle analysis to analyze anomaly-handling requirements in evolving systems.

Defect reports generated for faults found during testing provide a rich source of information regarding problematic phrases used in requirements documents. These reports indicate that faults often derive from instances of ambiguous, incorrect or otherwise deficient language. In this paper, we report on a method combining elements of linguistic theory and information retrieval to guide the discovery of problematic phrases throughout a requirements specification, using defect reports and correction requests generated during testing to seed our detection process. We found that phrases known from these materials to be problematic have occurrence properties in requirements documents that both allow the direction of resources to prioritize their correction, and generate insights characterizing more general locations of difficulty within the requirements. Our findings lead to some recommendations for more efficiently and effectively managing certain natural language issues in the creation and maintenance of requirements specifications.

Abstract Many product families are modest in the sense that they consist of a sequence of incremental products with, at any point in time, only a few distinct products available and minimal variations among the products. Such product families, nevertheless, are often large, complex systems, widely deployed, and possessing stringent safety and performance requirements. This paper describes a case study that tends to confirm the value of using a product‐line approach for the architectural design of a modest product family. The paper describes the process, design alternatives, and lessons learned, both positive and negative, from the architectural design of one such family of medical image analysis products. Realized benefits included identifying previously unrecognized common behavior and sets of features that were likely to change together, aligning the architecture with specific market needs and with the organization, and reducing unplanned dependencies. Most interesting were the unanticipated benefits, including decoupling the product‐family architecture from the order of implementation of features, and using the product‐family architecture as a ‘guiding star’ with subsequent releases moving toward, rather than away from, the planned architecture. Copyright © 2004 John Wiley & Sons, Ltd.

Analysis of anomalies that occur during operations is an important means of improving the quality of current and future software. Although the benefits of anomaly analysis of operational software are widely recognized, there has been relatively little research on anomaly analysis of safety-critical systems. In particular, patterns of software anomaly data for operational, safety-critical systems are not well understood. We present the results of a pilot study using orthogonal defect classification (ODC) to analyze nearly two hundred such anomalies on seven spacecraft systems. These data show several unexpected classification patterns such as the causal role of difficulties accessing or delivering data, of hardware degradation, and of rare events. The anomalies often revealed latent software requirements that were essential for robust, correct operation of the system. The anomalies also caused changes to documentation and to operational procedures to prevent the same anomalous situations from recurring. Feedback from operational anomaly reports helped measure the accuracy of assumptions about operational profiles, identified unexpected dependencies among embedded software and their systems and environment, and indicated needed improvements to the software, the development process, and the operational procedures. The results indicate that, for long-lived, critical systems, analysis of the most severe anomalies can be a useful mechanism both for maintaining safer, deployed systems and for building safer, similar systems in the future.

This paper describes the role of requirements discovery during the testing of a safety-critical software system. Analysis of problem reports generated by the integration and system testing of an embedded, safety-critical software system identified four common mechanisms for requirements discovery and resolution during testing: (1) Incomplete requirements, resolved by changes to the software, (2) Unexpected requirements interactions, resolved by changes to the operational procedures, (3) Requirements confusion by the testers, resolved by changes to the documentation, and (4) Requirements confusion by the testers, resolved by a determination that no change was needed The experience reported here confirms that requirements discovery during testing is frequently due to communication difficulties and subtle interface issues. The results also suggest that "false positive" problem reports from testing (in which the software behaves correctly but unexpectedly) provide a rich source of requirements information that can be used to reduce operational anomalies in critical systems.

We describe the results of an investigation into requirements discovery during testing and operations. Requirements discovery includes both new requirements and new knowledge regarding existing requirements. Analysis of anomaly reports shows that many of the anomalies that occur during these phases involve requirements discovery. Previous work identified four common mechanisms for requirements discovery and resolution during testing. The results reported here extend that work in two ways: (1) to show that very similar requirements-discovery mechanisms are at work in both testing and operations, and (2) to evaluate the requirements-discovery mechanisms against experience with seven additional systems. We discuss the consequences of these classifications and results in terms of reducing requirements-based defects in critical, embedded systems.

Software fault tree analysis (SFTA) provides a structured way to reason about the safety or reliability of a software system. As such, SFTA is widely used in mission-critical applications to investigate contributing causes to possible hazards or failures. In this paper we propose an approach similar to SFTA for product families. The contribution of the paper is to define a top-down, tree-based analysis technique, the fault contribution tree analysis (FCTA), that operates on the results of a product-family domain analysis and to describe a method by which the FCTA of a product family can serve as a reusable asset in the building of new members of the family. Specifically, we describe both the construction of the fault contribution tree for a product family (domain engineering) and the reuse of the appropriately pruned fault contribution tree for the analysis of a new member of the product family (application engineering). The paper describes several challenges to this approach, including evolution of the product family, handling of subfamilies, and distinguishing the limits of safe reuse of the FCTA, and suggests partial solutions to these issues as well as directions for future work. The paper illustrates the techniques with examples from applications to two product families.

This paper reports the results of a small study of requirements changes to the onboard software of three spacecraft subsequent to launch. Only those requirement changes that resulted from post-launch anomalies (i.e., during operations) were of interest here, since the goal was to better understand the relationship between critical anomalies during operations and how safety-critical requirements evolve. The results of the study were surprising in that anomaly-driven, post-launch requirements changes were rarely due to previous requirements having been incorrect. Instead, changes involved new requirements: (1) for the software to handle rare events; or (2) for the software to compensate for hardware failures or limitations. The prevalence of new requirements as a result of post-launch anomalies suggests a need for increased requirements-engineering support of maintenance activities in these systems. The results also confirm both the difficulty and the benefits of pursuing requirements completeness, especially in terms of fault tolerance, during development of critical systems.

This paper addresses the issue of how to perform architectural analysis on an existing product line architecture. The con tribution of the paper is to identify and demonstrate a repeatable product line architecture analysis process. The approach defines a “good” product line architecture in terms of those quality attributes required by the particular product line under development. It then analyzes the architecture against these criteria by both manual and tool-supported methods. The phased approach described in this paper provides a structured analysis of an existing product line architecture using (1) formal specification of the high-level architecture, (2) manual analysis of scenarios to exercise the architecture's support for required variabilities, and (3) model checking of critical behaviors at the architectural level that are required for all systems in the product line. Results of an application to a software product line of spaceborne telescopes are used to explain and evaluate the approach.

Current needs for high-reliability reusable software, rapid evolutionary development and verification of innovative software architectures have focused attention on improving techniques for analysing the safety and reliability of embedded software. The work reported in this paper integrates two successful safety analysis techniques which have been used separately on software and hardware into the system engineering process. This process combines SFMECA (software failure modes and effects criticality analysis) and SFTA (software fault tree analysis) in a way that can be readily adapted to a particular project's evolving system needs. The technique was used on two recent space instruments: the Mars Microprobe Project and the Earth Orbiting System's Microwave Limb Sounder. The main lessons learned from this experience are discussed: (1) flexible use, (2) a risk-driven rather than sequential approach, (3) "zoom-in/zoom-out" use, (4) SFMECA and SFTA as complementary techniques, (5) preserving traceability, and (6) applicability to fault protection software.

The paper describes three case studies in the lightweight application of formal methods to requirements modeling for spacecraft fault protection systems. The case studies differ from previously reported applications of formal methods in that formal methods were applied very early in the requirements engineering process to validate the evolving requirements. The results were fed back into the projects to improve the informal specifications. For each case study, we describe what methods were applied, how they were applied, how much effort was involved, and what the findings were. In all three cases, formal methods enhanced the existing verification and validation processes by testing key properties of the evolving requirements and helping to identify weaknesses. We conclude that the benefits gained from early modeling of unstable requirements more than outweigh the effort needed to maintain multiple representations.

The root causes of safety-related software errors in safety-critical embedded systems are analyzed. The results show that software errors identified as potentially hazardous to the system tend to be produced by different error mechanisms than those that produce nonsafety-related software errors. Safety-related software errors are shown to arise most commonly from: discrepancies between the documented requirements specifications and the requirements needed for correct functioning of the system; and misunderstandings of the interface of the software with the rest of the system. These results are used to identify methods by which requirements errors can be prevented. The goal is to reduce safety-related software errors and to enhance the safety of complex, embedded systems.>

This paper provides a Safety Checklist for use during the analysis of software requirements for spacecraft and others safety-critical, embedded systems. The checklist specifically targets the two most common causes of safety-related software errors: (1) inadequate interface requirements and (2) discrepancies between the documented requirements and the requirements actually needed for correct functioning of the system. The analysis criteria represented in the checklist are evaluated by application to two spacecraft projects. Use of the checklist to enhance the software-requirements analysis is shown to reduce the number of safety-related software errors.

A mechanism for modeling timing, precedence, and data-consistency constraints on concurrently executing processes is presented. The model allows durations and intervals between events to be specified. An algorithm is provided to detect schedules which may be unsafe with respect to the constraints. This work, motivated by the design and validation of autonomous error-recovery strategies on the Galileo spacecraft, appears to be applicable to a variety of asynchronous real-time systems.>