This work develops a new basis for evaluating the reliability benefits of diverse software, based on fault injection testing. In particular, the work investigates new forms of argumentation that could in principle be used to justify diversity as a basis for the construction of safety claims. Failure distributions of two versions of diverse software under various fault conditions are revealed separately by fault injection methods, and then the common failure probability of the version-pair can be estimated. The approach is justified theoretically, and cross validated with other work. This method is also used to explain the fundamental influence of failure distributions on diversity. Furthermore, the unique capabilities of the method are demonstrated by implementation of the fault injection test on a program pair.

Dependability assessment of safety-critical or safety-related software components is an important issue for example within the nuclear industry, the avionics sector or the military. Statistical testing is one way of quantifying the dependability of a given software product. The use of sector-specific standards with their suggested test-criteria is another (nonquantitative) way of aiming at employing only components that are "dependable enough". Ideally, both, the acknowledged test criteria and statistical test methods should come into play when assessing software dependability. We want to - in the long-term - move towards this aim. Thus we investigate in this paper a model to combine the fault-detection power of a given test-set (a test-adequacy criterion) with the statistical power of the test-set, i.e. the number of statistical tests within the test-set. With this model we aim at drawing out of any given test-set - whether devised by a plant engineer or a statistician - the overall contribution it can make to dependability assessment.

We address the difficult problem of estimating the reliability of multiple-version software. The central issue is the degree of statistical dependence between failures of diverse versions. Previously published models of failure dependence described what behavior could be expected "on average" from a pair of "independently generated" versions. We focus instead on predictions using specific information about a given pair of versions. The concept of "variation of difficulty" between situations to which software may be subject is central to the previous models cited, and it turns out to be central for our question as well. We provide new understanding of various alternative imprecise estimates of system reliability and some results of practical use, especially with diverse systems assembled from pre-existing (e.g., "off-the-shelf") subsystems. System designers, users, and regulators need useful bounds on the probability of system failure. We discuss how to use reliability data about the individual diverse versions to obtain upper bounds and other useful information for decision making. These bounds are greatly affected by how the versions' probabilities of failure vary between subdomains of the demand space or between operating regimes-it is even possible in some cases to demonstrate, before operation, upper bounds that are very close to the true probability of failure of the system-and by the level of detail with which these variations are documented in the data.

An important problem for computer-based systems is providing fault tolerance for unknown (at the time of commencement of service) systematic design errors. Such design errors can have a long latency in normal operation and only become apparent under specific conditions associated with particular combinations of input and internal system states. The use of 'diverse' software versions remains a possible approach to prevent coincidental failure, but its potential value has never been quantified. This paper presents the application of data-flow and constant perturbation to simulate the introduction of faults or errors into programs and explores methods to establish the magnitudes and locations of the associated input space failure regions. Used together, these two techniques enable failure behaviour to be described in a quantitative way and provide a method to estimate the diversity of multi-version software. A simple case and a industrial software are studied to illustrate the applications of the approach.

Following the current trend towards the increased use of commercial off-the-shelf (COTS) components in safety-critical systems, the need arises to address safety issues related to COTS-based systems. In this paper, we introduce a hierarchical model to estimate the probability of failure, on demand, of a software system consisting of components. Thereby, available evidence on the components' failure behaviour is combined using a Bayesian approach. An additive error term is proposed to model the changes in a component's prior reliability once the component is transferred from the operational environment assumed by the software developer into the actual operating environment. Statistical system test data are incorporated to calculate the estimator of the system failure rate. This approach involves prior knowledge at the level where it is most likely to be available: the component level.

A statistical software testing model is proposed in which white box factors have a role. The model combines test adequacy notions with statistical analysis, and in so doing provides a rudimentary treatment of dependencies between test results caused by the execution of common code during the tests. The model is used to estimate the probability of failure on demand for software performing safety shutdown functions on large plants and concerns the case where extensive test results are available on the latest version of the software, none of which have resulted in software failure. According to the model, there are circumstances in which some current statistical models for dynamic software testing are too conservative, and others are not conservative, depending on the software architecture.>

Abstract The term ‘inductive inference’ denotes the process of hypothesizing a general rule from examples. It can be considered as the inverse process of program testing, which is a process of sampling the behaviour of a program and gathering confidence in the quality of the software from the samples. As one of the fundamental and ubiquitous components of intelligent behaviour, much effort has been spent on both the theory and practice of inductive inference as a branch of artificial intelligence. In this paper, software testing and inductive inference are reviewed to illustrate how the rich and solid theory of inductive inference can be used to study the foundations of software testing.