Numerous database replication schemes are built on the crash failure assumption where majority of failures are self-evident as defined in [1]. The study in [1] convincingly refuted this common assumption showing that many of the faults in relational Database Management Systems (DBMSs) cause systematic non-crash failures. Similar results were obtained in the subsequent study [2]. Consequently, the existing database replication solutions, which typically use the same DBMS, are ineffective fault-tolerant mechanisms. Conversely, using diverse DBMSs is a suitable way of protecting against non-crash failures. We have built a middleware-based database replication protocol, DivRep, and deployed it with diverse database servers (DivSQL), for improved fault tolerance. DivSQL provides strict Snapshot Isolation (SI) guarantees, and assumes “incorrect results” failure model (IRFM) – the most realistic one based on the extensive experimental analyses of DBMS faults ([1], [2]). The dependability gain comes with the inherent performance overhead. We provide a comprehensive performance evaluation of DivSQL using 3 diverse DBMSs (two are leaders in the field).

This paper offers a new approach to modelling the effect of cyber-attacks on reliability of software used in industrial control applications. The model is based on the view that successful cyber-attacks introduce failure regions, which are not present in non-compromised software. The model is then extended to cover a fault tolerant architecture, such as the 1-out-of-2 software, popular for building industrial protection systems. The model is used to study the effectiveness of software maintenance policies such as patching and "cleansing" ("proactive recovery") under different adversary models ranging from independent attacks to sophisticated synchronized attacks on the channels. We demonstrate that the effect of attacks on reliability of diverse software significantly depends on the adversary model. Under synchronized attacks system reliability may be more than an order of magnitude worse than under independent attacks on the channels. These findings, although not surprising, highlight the importance of using an adequate adversary model in the assessment of how effective various cyber-security controls are.

Use of diverse software components is a viable defence against common-mode failures in redundant software-based systems. Various forms of "Diversity-Seeking Decisions" ("DSDs") can be applied to the process of developing, or procuring, redundant components, to improve the chances of the resulting components not failing on the same demands. An open question is how effective these decisions, and their combinations, are for achieving large enough reliability gains. Using a large population of software programs, we studied experimentally the effectiveness of specific "DSDs" (and their combinations) mandating differences between redundant components. Some of these combinations produced much better improvements in system probability of failure per demand (PFD) than "uncontrolled" diversity did. Yet, our findings suggest that the gains from such "DSDs" vary significantly between them and between the application problems studied. The relationship between DSDs and system PFD is complex and does not allow for simple universal rules (e.g. "the more diversity the better") to apply.

The most popular forms of fault tolerance against design faults use "asymmetric" architectures in which a "primary" part performs the computation and a "secondary" part is in charge of detecting errors and performing some kind of error processing and recovery. In contrast, the most studied forms of software fault tolerance are "symmetric" ones, e.g. N-version programming. The latter are often controversial, the former are not. We discuss how to assess the dependability gains achieved by these methods. Substantial difficulties have been shown to exist for symmetric schemes, but we show that the same difficulties affect asymmetric schemes. Indeed, the latter present somewhat subtler problems. In both cases, to predict the dependability of the fault-tolerant system it is not enough to know the dependability of the individual components. We extend to asymmetric architectures the style of probabilistic modeling that has been useful for describing the dependability of "symmetric" architectures, to highlight factors that complicate the assessment. In the light of these models, we finally discuss fault injection approaches to estimating coverage factors. We highlight the limits of what can be predicted and some useful research directions towards clarifying and extending the range of situations in which estimates of coverage of fault tolerance mechanisms can be trusted.

If an off-the-shelf software product exhibits poor dependability due to design faults, then software fault tolerance is often the only way available to users and system integrators to alleviate the problem. Thanks to low acquisition costs, even using multiple versions of software in a parallel architecture, which is a scheme formerly reserved for few and highly critical applications, may become viable for many applications. We have studied the potential dependability gains from these solutions for off-the-shelf database servers. We based the study on the bug reports available for four off-the-shelf SQL servers plus later releases of two of them. We found that many of these faults cause systematic noncrash failures, which is a category ignored by most studies and standard implementations of fault tolerance for databases. Our observations suggest that diverse redundancy would be effective for tolerating design faults in this category of products. Only in very few cases would demands that triggered a bug in one server cause failures in another one, and there were no coincident failures in more than two of the servers. Use of different releases of the same product would also tolerate a significant fraction of the faults. We report our results and discuss their implications, the architectural options available for exploiting them, and the difficulties that they may present.

Database replication is widely used to improve both fault tolerance and DBMS performance. Non-diverse database replication has a significant limitation - it is effective against crash failures only. Diverse redundancy is an effective mechanism of tolerating a wider range of failures, including many non-crash failures. However it has not been adopted in practice because many see DBMS performance as the main concern. In this paper we show experimental evidence that diverse redundancy (diverse replication) can bring benefits in terms of DBMS performance, too. We report on experimental results with an optimistic architecture built with two diverse DBMSs under a load derived from TPC-C benchmark, which show that a diverse pair performs faster not only than non-diverse pairs but also than the individual copies of the DBMSs used. This result is important because it shows potential for DBMS performance better than anything achievable with the available off-the-shelf servers

Fault tolerance is often the only viable way of obtaining the required system dependability from systems built out of "off-the-shelf" (OTS) products. We have studied a sample of bug reports from four off-the-shelf SQL servers so as to estimate the possible advantages of software fault tolerance - in the form of modular redundancy with diversity - in complex off-the-shelf software. We checked whether these bugs would cause coincident failures in more than one of the servers. We found that very few bugs affected two of the four servers, and none caused failures in more than two. We also found that only four of these bugs would cause identical, undetectable failures in two servers. Therefore, a fault-tolerant server, built with diverse off-the-shelf servers, seems to have a good chance of delivering improvements in availability and failure rates compared with the individual off-the-shelf servers or their replicated, nondiverse configurations.

Previous models have investigated the impact upon diversity - and hence upon the reliability of fault-tolerant software built from 'diverse' versions - of the variation in 'difficulty' of demands over the demand space. These models are essentially static, taking a single snapshot view of the system. In this paper, we consider a generalisation in which the individual versions are allowed to evolve - and their reliability to grow - through debugging. In particular, we examine the trade-off that occurs in testing between, on the one hand, the increasing reliability of individual versions, and on the other hand the possible diminution of diversity.

We address the difficult problem of estimating the reliability of multiple-version software. The central issue is the degree of statistical dependence between failures of diverse versions. Previously published models of failure dependence described what behavior could be expected "on average" from a pair of "independently generated" versions. We focus instead on predictions using specific information about a given pair of versions. The concept of "variation of difficulty" between situations to which software may be subject is central to the previous models cited, and it turns out to be central for our question as well. We provide new understanding of various alternative imprecise estimates of system reliability and some results of practical use, especially with diverse systems assembled from pre-existing (e.g., "off-the-shelf") subsystems. System designers, users, and regulators need useful bounds on the probability of system failure. We discuss how to use reliability data about the individual diverse versions to obtain upper bounds and other useful information for decision making. These bounds are greatly affected by how the versions' probabilities of failure vary between subdomains of the demand space or between operating regimes-it is even possible in some cases to demonstrate, before operation, upper bounds that are very close to the true probability of failure of the system-and by the level of detail with which these variations are documented in the data.

Design diversity is a defence against design faults causing common-mode failure in redundant systems, but we badly lack knowledge about how much reliability it will buy in practice, and thus about its cost-effectiveness, the situations in which it is an appropriate solution and how it should be taken into account by assessors and safety regulators. Both current practice and the scientific debate about design diversity depend largely on intuition. More formal probabilistic reasoning would facilitate critical discussion and empirical validation of any predictions: to this aim, we propose a model of the generation of faults and failures in two separately-developed program versions. We show results on: (i) what degree of reliability improvement an assessor can reliably expect from diversity; and (ii) how this reliability improvement may change with higher-quality development processes. We discuss the practical relevance of these results and the degree to which they can be trusted.

Considers what happens when several different fault-finding techniques are used together. The effectiveness of such multi-technique approaches depends upon a quite subtle interplay between their individual efficacies. The modeling tool we use to study this problem is closely related to earlier work on software design diversity which showed that it would be unreasonable even to expect software versions that were developed truly independently to fail independently of one another. The key idea was a "difficulty function" over the input space. Later work extended these ideas to introduce a notion of "forced" diversity. In this paper, we show that many of these results for design diversity have counterparts in diverse fault detection in a single software version. We define measures of fault-finding effectiveness and diversity, and show how these might be used to give guidance for the optimal application of different fault-finding procedures to a particular program. The effects on reliability of repeated applications of a particular fault-finding procedure are not statistically independent; such an incorrect assumption of independence will always give results that are too optimistic. For diverse fault-finding procedures, it is possible for effectiveness to be even greater than it would be under an assumption of statistical independence. Diversity of fault-finding procedures is a good thing and should be applied as widely as possible. The model is illustrated using some data from an experimental investigation into diverse fault-finding on a railway signalling application.

Some experimental results related to the effect of testing on the reliability of software systems are presented. The experiments were performed on a simulator designed for behavioral modelling of software systems in the presence of design faults. The topics of the experiments are defined as: the influence of the testing profile on the reliability evolution of a single version system, the influence of the design faults size on the evolution of system reliability in different testing profiles and the effect of the testing strategy on the reliability of 1-out-of-2 fault-tolerant scheme. In the last topic, three testing strategies are considered: individual, in coalition and back-to-back testing.