SUMMARY Many papers have been published on analysis and prediction of software faults and/or failures, but few addressed the software fixes made to correct the faults and prevent failures from reoccurring. This paper contributes towards filling this gap by focusing on empirical characterization of software fixes. The results are based on the data extracted from a safety–critical NASA mission. In particular, 21 large‐scale software components (which together constitute over 8000 files and millions of lines of code) were analysed. The unique characteristic of this work is the fact that links were established from software faults (i.e. the root causes) to (potential or observed) failures and consequently to fixes made to correct these faults. Specifically, for the fixes associated with individual failures, the spread across software components and types of software artifacts being fixed was studied. Our results showed that significant number of software failures required fixes in multiple software components and/or multiple software artifacts (i.e. 15% and 26%, respectively). The results also showed that the patterns of software components that were often fixed together were significantly affected by the software architecture. Furthermore, the types of fixed software artifacts were highly correlated with fault type and they had different distributions for prerelease and post‐release failures. Copyright © 2013 John Wiley & Sons, Ltd.

The assumption of independence among successive software runs, common to many software reliability models, often is a simplification of the actual behavior. This paper addresses the problem of estimating software reliability when the successive software runs are statistically correlated, that is, when an outcome of a run depends on one or more of its previous runs. First, we propose a generalization of our previous work using higher order Markov chain to model a sequence of dependent software runs. Then, we conduct an empirical study for exploring the phenomenon of dependent software runs using three software applications as case studies. Based on two statistical approaches, we show that the outcomes of software runs (i.e., success or failure) for two of the case studies are dependent on the outcome of one or more previous runs, in which case first or higher order Markov chain models are appropriate. Finally, we estimate the parameters of the appropriate models and discuss the effects of dependent software runs on the estimates of the software reliability.

The benefits of the analysis of software faults and failures have been widely recognized. However, detailed studies based on empirical data are rare. In this paper, we analyze the fault and failure data from two large, real-world case studies. Specifically, we explore: 1) the localization of faults that lead to individual software failures and 2) the distribution of different types of software faults. Our results show that individual failures are often caused by multiple faults spread throughout the system. This observation is important since it does not support several heuristics and assumptions used in the past. In addition, it clearly indicates that finding and fixing faults that lead to such software failures in large, complex systems are often difficult and challenging tasks despite the advances in software development. Our results also show that requirement faults, coding faults, and data problems are the three most common types of software faults. Furthermore, these results show that contrary to the popular belief, a significant percentage of failures are linked to late life cycle activities. Another important aspect of our work is that we conduct intra- and interproject comparisons, as well as comparisons with the findings from related studies. The consistency of several main trends across software systems in this paper and several related research efforts suggests that these trends are likely to be intrinsic characteristics of software faults and failures rather than project specific.

Uncertainty analysis through sensitivity studies and quantification of the variance of the reliability estimate has become more common in architecture-based software reliability studies. However, up to this point no attempts have been made to explicate the results of such analysis. Our earlier work based on several medium to large scale empirical studies showed that a very few parameters have a significant impact on the variability of system reliability. This paper explains the reasons behind this phenomenon. Unlike related work that considered the impact of the parameters on software reliability either through their model sensitivity or through uncertainty of their estimates, we consider both. Furthermore, we look at all parameters, i.e., components reliabilities and probabilities of transfer of control between components. Based on theoretical and empirical arguments, we justify why a few parameters contribute most of the variance of the reliability estimate. Comparing our results with those obtained through simple model sensitivity studies shows that such studies are not always sufficient to accurately quantify the impact of critical components on variability of system reliability.

Our earlier research work on applying architecture-based software reliability models on a large scale case study allowed us to test how and when they work, to understand their limitations, and to outline the issues that need future research. In this paper we first present an additional case study which confirms our earlier findings. Then, we present uncertainty analysis of architecture-based software reliability for both case studies. The results show that Monte Carlo method scales better than the method of moments. The sensitivity analysis based on Monte Carlo method shows that (1) small number of parameters contribute to the most of the variation in system reliability and (2) given an operational profile, components' reliabilities have more significant impact on system reliability than transition probabilities. Finally, we summarize the lessons learned from conducting large scale empirical case studies for the purpose of architecture-based reliability assessment and uncertainty analysis.

In this paper we present an empirical study of architecture-based software reliability based on a large open source application which consists of 350,000 lines of C code. The goals of our study are to analyze empirically the adequacy, applicability, and accuracy of architecture-based software reliability models. For this purpose we developed innovative approaches to efficiently extract and more accurately analyze a large amount of empirical data. Applying the theoretical results on a large scale field study allows us to test how and when they work, to understand their limitations, and outline the issues that need attention in the future research studies. Thus, our results show that for a subset of failures which can clearly be attributed to single components, both the composite and hierarchical models are very accurate when compared to the actual reliability. However, the assumptions made by the existing architecture-based software reliability models do not allow accounting for the remaining failures which led to fixing faults in multiple components. These results show that in order to progress further, software reliability engineering should go through cycles of building models, testing them empirically, learning from the experiments, and refining the models to capture the newly discovered phenomena.