Many systems make use of concurrent tasks, however it is often difficult to test concurrent design. Therefore, many test cases are simplified and do not fully test all concurrency aspects of the system. We encountered this problem when analyzing test cases for concurrent flight software at NASA. To address this problem, we developed and evaluated a model based testing (MBT) technique for testing of concurrent systems. Using MBT, the tester creates a model, which is based on the requirements of the system under test (SUT), and lets the computer generate innumerable test cases automatically from the model. We evaluate the effectiveness of the technique using Microsoft's Spec Explorer MBT tool. We apply the technique on NASA's Core Flight Software (cFS) software bus module API, which is based on a concurrent publisher-subscriber architecture style and is a safety-critical system. We describe how we created a test automation architecture for testing concurrent inter-task communication as carried out by the software bus. We also investigate the type of issues the technique for testing of concurrent systems can find as well as what degree of code coverage it can achieve.

Testing is necessary for all types of systems, but becomes difficult when the tester cannot easily determine whether the system delivers the correct result or not. NASA's Data Access Toolkit allows NASA analysts to query a large database of telemetry data. Since the user is unfamiliar with the data and several data transformations can occur, it is impossible to determine whether the system behaves correctly or not in full scale production situations. Small scale testing was already conducted manually by other teams and unit testing was conducted on individual functions. However, there was still a need for full scale acceptance testing on a broad scale. We describe how we addressed this testing problem by applying the idea of metamorphic testing [1]. Specifically, we base it on equivalence of queries and by using the system itself for testing. The approach is implemented using a model-based testing approach in combination with a test data generation and test case outcome analysis strategy. We also discuss some of the issues that were detected using this approach.

This article introduces the Architecture Discovery and Analysis Method (ADAM). ADAM supports the discovery of module and runtime views as well as the analysis of quality attributes, such as testability, performance, and maintainability, of software systems. The premise of ADAM is that the implementation constructs, architecture constructs, concerns, and quality attributes are all influenced by the external entities (e.g., libraries, frameworks, COTS software) used by the system under analysis. The analysis uses such external dependencies to identify, classify, and review a minimal set of key source-code files supported by a knowledge base of the external entities. Given the benefits of analyzing external dependencies as a way to discover architectures and potential risks, it is demonstrated that dependencies to external entities are useful not only for architecture discovery but also for analysis of quality attributes. ADAM is evaluated using the NASA's Space Network Access System (SNAS). The results show that this method offers systematic guidelines for discovering the architecture and locating potential risks (e.g., low testability and decreased performance) that are hidden deep inside the system implementation. Some generally applicable lessons for developers and analysts, as well as threats to validity are also discussed.

Exchangeability between software components such as operating systems, middleware, databases, and hardware components is a common requirement in many software systems. One way to enable exchangeability is to promote indirect use through a common interface and an implementation for each component that wraps the original component. As developers use the interface instead of the underlying component, they assume that the software system will behave in a specific way independently of the actual component in use. However, differences in the implementations of the wrappers may lead to different behavior when one component is changed for another, which might lead to failures in the field. This work reports on a simple, yet effective approach to detect these differences. The approach is based on tool-supported reviews leveraging lightweight static analysis and machine learning. The approach is evaluated in a case study that analyzes NASA's Operating System Abstraction Layer (OSAL), which is used in various space missions. We detected 84 corner-case issues of which 57 turned out to be bugs that could have resulted in runtime failures.

We present a case study that evaluates the applicability and effectiveness of model-based testing in detecting bugs in real-world, mission-critical systems. NASA's Operating System ion Layer (OSAL) is the subject system of this paper. The OSAL is a reusable framework that wraps several operating systems (OS) and is used extensively in NASA's flight software missions. We developed a suite of behavioral models, represented as hierarchical finite state machines (FSMs), of the core file system API and generated a large number of test cases automatically. We then automatically executed these test cases against the OSAL. The results show that the OSAL is a high quality product. Naturally, due to the systematic and rigorous nature of MBT, we detected a few previously unknown “corner-case” bugs and issues, which escaped traditional manual testing and code reviews. We discuss the MBT architecture, the detected bugs, the code coverage of generated tests, as well as threats to validity of the study.

New research is underway at the FDA to investigate the benefits of integrating architecture analysis into safety evaluations of medical-device software. Due to the complexity in setting up testing environments for such software, the FDA is unable to conduct large-scale safety testing, instead, it must rely on other techniques to build an argument for whether the software is safe or not. The architecture analysis approach, formalized using relational algebra, is based on reconstructing abstract, yet precise, architectural views from source code to help build such arguments about safety. This paper discusses the use of the formal approach to analyze the Computer-Assisted Resuscitation Algorithm (CARA) software, which controls an infusion pump designed to provide automated assistance for transfusing blood. The results suggest that a) architecture analysis offers many insights related to software quality in general and testability (i.e., the ease of testing) and its impact on safety in particular, and b) architectural analysis results can be used to help configure static analysis tools to improve their performance for verifying safety properties.

Quality assurance in general and testing in particular remains as an unsolved problem in product line engineering. It is unclear, from an economic point of view, whether one should test each product line member separately or test just product-specific parts and compose them with tested core assets from family engineering. While the former strategy introduces redundancy in testing, the latter strategy is also more difficult because of genericity. Ideally, we need to carefully calculate cost-benefits of both cases and define a test strategy accordingly. Currently, there is no economic model for testing in the context of product lines. Thus, we introduce an initial economic model, and then perform Monte-Carlo simulations to calculate cost-benefit of the two test strategies. For the Testo product line of climate and flue gas measurement devices, the results showed that, using an infrastructure-focused test strategy, on average a cost saving of 13% can be expected with 87% certainty.

Engine Control Systems (ECS) for automobiles have numerous variants for many manufactures and different markets. To improve development efficiency, exploiting ECS commonalities and predicting their variability are mandatory. The concept of software product line engineering meets the business background of ECS. However, we should carefully investigate the expected technical, economical, and organizational effects of introducing this strategy into existing products.This paper explains an approach for assessing the potential of merging existing embedded software into a product line approach. The definition of an economically useful product line approach requires two things: analyzing return on investment (ROI) expectations of a product line and understanding the effort required for building reusable assets. We did a clone analysis to provide the basis for effort estimation for merge potential assessment of existing variants. We also report on a case study with ECS. We package the lessons learned and open issues that arose during the case study.

The decision of an organization to introduce product line engineering depends on a sound and careful analysis of risks and return on investment. The latter is computed by an economic model, which relies on high quality input and must reflect the envisioned migration strategy sufficiently. To facilitate risk analysis, this paper applies Monte-Carlo simulation to an existing product line economic model. Additionally, the model is extended by the support of product line generations that is, considering the degeneration of product line infrastructures and taking reinvestment into an existing product line into account. The practical application of the model is demonstrated by an industrial case study

This article reports on the experience of monitoring the evolution of a product family of financial Web service software system with OO-metrics over a period of three years. Every four to six months, a snapshot of the system was taken and a set of OO metrics was computed. The classes with extreme metric values or important value changes became the focus of analysis and discussions among the developers, designers, and managers. This way, potential problems were identified and corrected early.