In recent years we have become interested in the problem of assessing the probability of perfection of software-based systems which are sufficiently simple that they are "possibly perfect". By "perfection" we mean that the software of interest will never fail in a specific operating environment. We can never be certain that it is perfect, so our interest lies in claims for its probability of perfection. Our approach is Bayesian: our aim is to model the changes to this probability of perfection as we see evidence of failure-free working. Much of the paper considers the difficult problem of expressing prior beliefs about the probability of failure on demand (pfd), and representing these mathematically. This requires the assessor to state his prior belief in perfection as a probability, and also to state what he believes are likely values of the pfd in the event that the system is not perfect. We take the view that it will be impractical for an assessor to express these beliefs as a complete distribution for pfd. Our approach to the problem has three threads. Firstly we assume that, although he cannot provide a full probabilistic description of his uncertainty in a single distribution, the assessor can express some precise but partial beliefs about the unknowns. Secondly, we assume that in the inevitable presence of such incompleteness, the Bayesian analysis needs to provide results that are guaranteed to be conservative (because the analyses we have in mind relate to critical systems). Finally, we seek to prune the set of prior distributions that the assessor finds acceptable in order that the conservatism of the results is no greater than it has to be, i.e. we propose, and eliminate, sets of priors that would appear generally unreasonable. We give some illustrative numerical examples of this approach, and note that the numerical values obtained for the posterior probability of perfection in this way seem potentially useful (although we make no claims for the practical realism of the numbers we use). We also note that the general approach here to the problem of expressing and using limited prior belief in a Bayesian analysis may have wider applicability than to the problem we have addressed.

In recent work, we have argued for a formal treatment of confidence about the claims made in dependability cases for software-based systems. The key idea underlying this work is "the inevitability of uncertainty": It is rarely possible to assert that a claim about safety or reliability is true with certainty. Much of this uncertainty is epistemic in nature, so it seems inevitable that expert judgment will continue to play an important role in dependability cases. Here, we consider a simple case where an expert makes a claim about the probability of failure on demand (pfd) of a subsystem of a wider system and is able to express his confidence about that claim probabilistically. An important, but difficult, problem then is how such subsystem (claim, confidence) pairs can be propagated through a dependability case for a wider system, of which the subsystems are components. An informal way forward is to justify, at high confidence, a strong claim, and then, conservatively, only claim something much weaker: "I'm 99 percent confident that the pfd is less than 10-5, so it's reasonable to be 100 percent confident that it is less than 10-3." These conservative pfds of subsystems can then be propagated simply through the dependability case of the wider system. In this paper, we provide formal support for such reasoning.

Society is increasingly requiring quantitative assessment of risk and associated dependability cases. Informally, a dependability case comprises some reasoning, based on assumptions and evidence, that supports a dependability claim at a particular level of confidence. In this paper we argue that a quantitative assessment of claim confidence is necessary for proper assessment of risk. We discuss the way in which confidence depends upon uncertainty about the underpinnings of the dependability case (truth of assumptions, correctness of reasoning, strength of evidence), and propose that probability is the appropriate measure of uncertainty. We discuss some of the obstacles to quantitative assessment of confidence (issues of composability of subsystem claims; of the multi-dimensional, multi-attribute nature of dependability claims; of the difficult role played by dependence between different kinds of evidence, assumptions, etc). We show that, even in simple cases, the confidence in a claim arising from a dependability case can be surprisingly low.

Fault tolerance via design diversity is often the only viable way of achieving sufficient dependability levels when using off-the-shelf components. We have reported previously on studies with bug reports of four open-source and commercial off-the-shelf database servers and later release of two of them. The results were very promising for designers of fault-tolerant solutions that wish to employ diverse servers: very few bugs caused failures in more than one server and none caused failure in more than two. In this paper we offer details of two approaches we have studied to construct reliability growth models for a 1-out-of-2 fault-tolerant server which utilize the bug reports. The models presented are of practical significance to system designers wishing to employ diversity with off-the-shelf components since often the bug reports are the only direct dependability evidence available to them.

The work described here concerns the use of so-called multilegged arguments to support dependability claims about software-based systems. The informal justification for the use of multilegged arguments is similar to that used to support the use of multiversion software in pursuit of high reliability or safety. Just as a diverse 1-out-of-2 system might be expected to be more reliable than each of its two component versions, so might a two-legged argument be expected to give greater confidence in the correctness of a dependability claim (for example, a safety claim) than would either of the argument legs alone. Our intention here is to treat these argument structures formally, in particular, by presenting a formal probabilistic treatment of "confidence,? which will be used as a measure of efficacy. This will enable claims for the efficacy of the multilegged approach to be made quantitatively, answering questions such as, "How much extra confidence about a system's safety will I have if I add a verification argument leg to an argument leg based upon statistical testing?? For this initial study, we concentrate on a simplified and idealized example of a safety system in which interest centers upon a claim about the probability of failure on demand. Our approach is to build a "Bayesian Belief Network? (BBN) model of a two-legged argument and manipulate this analytically via parameters that define its node probability tables. The aim here is to obtain greater insight than what is afforded by the more usual BBN treatment, which involves merely numerical manipulation. We show that the addition of a diverse second argument leg can indeed increase confidence in a dependability claim; in a reasonably plausible example, the doubt in the claim is reduced to one-third of the doubt present in the original single leg. However, we also show that there can be some unexpected and counterintuitive subtleties here; for example, an entirely supportive second leg can sometimes undermine an original argument, resulting, overall, in less confidence than what came from this original argument. Our results are neutral on the issue of whether such difficulties will arise in real life?that is, when real experts judge real systems.

Operational testing, which aims to generate sequences of test cases with the same statistical properties as those that would be experienced in real operational use, can be used to obtain quantitative measures of the reliability of software. In the case of safety critical software it is common to demand that all known faults are removed. This means that if there is a failure during the operational testing, the offending fault must be identified and removed. Thus an operational test for safety critical software takes the form of a specified number of test cases (or a specified period of working) that must be executed failure-free. This paper addresses the problem of specifying the numbers of test cases (or time periods) required for a test, when the previous test has terminated as a result of a failure. It has been proposed that, after the obligatory fix of the offending fault, the software should be treated as if it were completely novel, and be required to pass exactly the same test as originally specified. The reasoning here claims to be conservative, in as much as no credit is given for any previous failure-free operation prior to the failure that terminated the test. We show that, in fact, this is not a conservative approach in all cases, and propose instead some new Bayesian stopping rules. We show that the degree of conservatism in stopping rules depends upon the precise way in which the reliability requirement is expressed. We define a particular form of conservatism that seems desirable on intuitive grounds, and show that the stopping rules that exhibit this conservatism are also precisely the ones that seem preferable on other grounds.

Effects of shared use on the dependability of modular software are evaluated in terms of a generally defined stochastic model. The total system in question consists of a community of n users who share a software system with m modules. The input aspect of the operational environment, reflecting user demands at the module level, is represented by a continuous-time, finite-state Markov process, called the operational profile. The profile's construction is based on the isolated profiles of individual users which, in the case of heterogeneous use, are pairwise-distinct processes. Moreover, in the presence of other users, an individual profile can differ from its isolated version due to "slowdowns" caused by sharing. This multiuser profile is then combined with a failure model which, among other things, captures "stress" due to shared use. A number of basic issues are then addressed and resolved in terms of closed-form dependability solutions obtained for elementary 2-user, 1-module systems. Specifically, three measures are investigated in this manner, providing considerable insight into how dependability, as perceived by a subject user, is affected by the profile of an interfering user.

Ideally, a measure of the security of a system should capture quantitatively the intuitive notion of ‘the ability of the system to resist attack’. That is, it should be operational, reflecting the degree to which the system can be expected to remain