Over the past several years, we have been developing software fault predictors based on a system's measured structural evolution. We have previously shown significant linear relationships between code churn, a set of synthesized metrics, and the rate at which faults are inserted into the system in terms of number of faults per unit change in code churn. A limiting factor in this and other such investigations has been the absence of a quantitative, consistent and repeatable definition of what constitutes a fault. The rules for fault definition were not sufficiently rigorous to provide unambiguous, repeatable fault counts. Within the framework of a space mission software development effort at the Jet Propulsion Laboratory we have developed a standard for the precise enumeration of faults. This new standard permits software faults to be measured directly from configuration control documents. We compared the new method of counting faults with two existing techniques to determine whether the fault-counting technique has an effect on the quality of the fault models constructed from those counts. The new fault definition provides higher quality fault models than those obtained using the other definitions of fault. Copyright © 2006 John Wiley & Sons, Ltd.

Abstract Our current work involves developing methods of measuring changes to evolving software systems. We study a system's change characteristics over a large number of builds using the distinct sources of variation in the software metrics used to measure the system. We have been collaborating with a flight software technology development effort at the Jet Propulsion Laboratory (JPL) and have progressed in resolving the limitations of our earlier work in two distinct steps. First, we have developed a repeatable and consistent fault enumeration methodology, allowing them to be precisely and accurately measured. Second, we have developed a practical framework for automating fault measurement, which we applied to the JPL software system during its development. Every change to the system was measured and every identified fault was tracked to a specific code module. Our analysis indicates that measures of the evolving system's structure are strongly related to the number of faults inserted during its development, and that some types of change are more likely to result in the insertion of faults than others. The fault enumeration methodology ensures that the resulting fault model has greater predictive validity; it also provides a higher quality model than other popular definitions of a fault. Copyright © 2005 John Wiley & Sons, Ltd.

Over the past few years, we have been developing software fault predictors based on a system's measured structural evolution. We have previously shown there is a significant linear relationship between code chum, a set of synthesized metrics, and the rate at which faults are inserted into the system in terms of number of faults per unit change in code chum. A limiting factor in this and other investigations of a similar nature has been the absence of a quantitative, consistent, and repeatable definition of what constitutes a fault. The rules for fault definition were not sufficiently rigorous to provide unambiguous, repeatable fault counts. Within the framework of a space mission software development effort at the Jet Propulsion Laboratory (JPL) we have developed a standard for the precise enumeration of faults. This new standard permits software faults to be measured directly from configuration control documents. Our results indicate that reasonable predictors of the number of faults inserted into a software system can be developed from measures of the system's structural evolution. We compared the new method of counting faults with two existing techniques to determine whether the fault counting technique has an effect on the quality of the fault models constructed from those counts. The new fault definition provides higher quality fault models than those obtained using the other definitions of fault

Over the past several years, we have been developing methods of measuring the change characteristics of evolving software systems. Not all changes to software systems are equal. Some changes to these systems are very small and have low impact on the system as a whole. Other changes are substantial and have a very large impact of the fault proneness of the complete system. In this study we will identify the sources of variation in the set of software metrics used to measure the system. We will then study the change characteristics to the system over a large number of builds. We have begun a new investigation in these areas in collaboration with a flight software technology development effort at the Jet Propulsion Laboratory (JPL) and have progressed in resolving the limitations of the earlier work in two distinct steps. First, we have developed a standard for the enumeration of faults. This new standard permits software faults to be measured precisely and accurately. Second, we have developed a practical framework for automating the measurement of these faults. This new standard and fault measurement process was then applied to a software system's structural evolution during its development. Every change to the software system was measured and every fault was identified and tracked to a specific code module. The measurement process was implemented in a network appliance, minimizing the impact of measurement activities on development efforts and enabling the comparison of measurements across multiple development efforts. In this paper, we analyze the measurements of structural evolution and fault counts obtained from the JPL flight software technology development effort. Our results indicate that the measures of structural attributes of the evolving software system are suitable for forming predictors of the number of faults inserted into software modules during their development, and that some types of change are more likely to result in the insertion of faults than others. The new fault standard also insures that the model so developed has greater predictive validity.

Software systems change dramatically as they go through their various stages of development. From the first build of each such system to the last build the differences may be so great as to obscure the fact that it is still the same system. Developers commonly make this mistake when they talk about the system that they are developing. It might be referred to as the Management or whatever name seems to describe the software. This seems to imply that there is but one File Management System. The fact that is obscured when we talk about the File Management System is that today's build of the File Management System is probably vastly different in composition and in functionality from the original first-born File Management System of the first system build. We would like to be able to quantify the differences in the system from its first build, through all builds to the current one. Then and only then will it be possible to know how these systems have changed. The focus of my recent research in software maintenance has been in the area of measurement of software evolution. This research has led to some interesting empirical results that have direct implications for Lehman's eight laws of software evolution. Not all aspects of these eight laws of software evolution lend themselves to direct measurement. In particular the First Law suggests that a system must be continually adapted. This law is driven by forces outside of the software development arena that are very difficult to quantify. The third law is very similar from a measurement perspective. Self regulation is again a property that is attributable to outside forces. The 8th law is confined to the feedback systems nature of the processes controlling software evolution. This, again, is outside the purview of direct measurement. The remaining five laws do lend themselves to direct measurement from the products of software evolution. The main problem with measuring evolving software is that software systems have multiple attribute domains that are changing simultaneously. The size of a program changes. The data structures of that program change. The control complexity changes. In essence, measuring the software evolution process is a multivariate problem. The measurement of an evolving software system through the shifting sands of time is not an easy task. Perhaps one of the most difficult issues relates to the establishment of a baseline against which the evolving systems may be compared. When a number of successive system builds are to be measured, we will choose one of the systems as a baseline system. All others will be measured in relation to the chosen system. Sometimes it will be useful to select the initial system build for this baseline. If we select this system, then the measurements on all other systems will be taken in relation to the initial system configuration. Within this context, laws 4, 5, and 6 lend themselves directly to measurement directly from a baseline system. Data from the several projects at the Jet Propulsion Laboratory support this conjecture. Until very recently, it would have been very difficult to

An important aspect of developing models relating the number and type of faults in a software system to a set of structural measurement is defining what constitutes a fault. By definition, a fault is a structural imperfection in a software system that may lead to the system's eventually failing. A measurable and precise definition of what faults are makes it possible to accurately identify and count them, which in turn allows the formulation of models relating fault counts and types to other measurable attributes of a software system. Unfortunately, the most widely-used definitions are not measurable; there is no guarantee that two different individuals looking at the same set of failure reports and the same set of fault definitions will count the same number of underlying faults. The incomplete and ambiguous nature of current fault definitions adds a noise component to the inputs used in modeling fault content. If this noise component is sufficiently large, any attempt to develop a fault model will produce invalid results. As part of our on-going work in modeling software faults, we have developed a method of unambiguously identifying and counting faults. Specifically, we base our recognition and enumeration of software faults on the grammar of the language of the software system. By tokenizing the differences between a version of the system exhibiting a particular failure behavior, and the version in which changes were made to eliminate that behavior, we are able to unambiguously count the number of faults associated with that failure. With modern configuration management tools, the identification and counting of software faults can be automated.

Modern intrusion detection systems are comprised of three basically different approaches, host based, network based, and a third relatively recent addition called procedural based detection. The first two have been extremely popular in the commercial market for a number of years now because they are relatively simple to use, understand and maintain. However, they fall prey to a number of shortcomings such as scaling with increased traffic requirements, use of complex and false positive prone signature databases, and their inability to detect novel intrusive attempts. The procedural based intrusion detection, systems represent a great leap forward over current security technologies by addressing these and other concerns. This paper presents an overview of our work in creating a true procedural Disallowed Operational Anomaly (DOA) system.

Learning from software failures is an essential step towards the development of more reliable software systems and processes. However, as more intricate software systems are developed, determining the nature and causes of a software failure becomes a great challenge. Although many existing techniques can help in understanding the nature of the failure, they are limited in some of the following aspects. First, they work only within controlled environments. Second, they have a major impact on the target system behavior. Third, they assume that a failure can be reproduced. Fourth, they lack enough support to carry out a structured failure analysis. The authors present the Software Black Box (SBB) as an alternative mechanism for failure investigation. The SBB is different from its predecessors in that it was specifically designed to be embedded in a target system and assist in the investigation of failures by reconstructing the events that lead to the failure. The SBB architecture is discussed, and a set of failure scenarios that reveal the SBB potential in assisting failure investigation is presented.

The test process for evolving software systems takes on a different measurement aspect than that of new systems. Existing systems are generally being modified on a continuing basis as a normal part of the software maintenance activity. This process of product modifications is fault prone because faults are introduced in the code as it is being modified. From a statistical perspective, regression testing should be focused on those areas that are most likely to contain the introduced faults. Based on that premise, we have developed an evolutionary fault index that works as a fault surrogate varying in the same manner as faults. However, the knowledge as to the location of probable faults is not enough to assess the capabilities of a regression test suite. It is necessary to understand what the software is doing under each test. For that purpose, test execution profiles are gathered. Test execution profiles and the evolutionary fault indexes are combined in one methodology to provide an assessment of the overall regression testing activity and the suitability of each individual test. The methodology is illustrated with data from a 300 KLOC embedded system and its corresponding regression test suite. Copyright © 2000 John Wiley & Sons, Ltd..

A great deal has been contributed to the body of literature in software engineering recently on the subject of software development processes. Unfortunately, missing from the software engineering literature is the notion of software measurement process. Somehow, measurement is something that you do once on a piece of software. These measurement data are then given to the software quality people who manage to misuse these data, draw the wrong conclusions about the meaning of the data, write reports about the data, and who are ultimately ignored by all concerned with the software development process. At the other end of this measurement data continuum is the university professor who takes these data of unknown value and origin from the software quality staff and proceeds to develop models of software development, software quality, or whatever else pleases his/her fancy.

This study presents a methodology that will produce a viable fault surrogate. The focus of the effort is on the precise measurement of software development process and product outcomes. Tools and processes for the static measurement of the source code have been installed and made operational in a large embedded software system. Source code measurements have been gathered unobtrusively for each build in the software evolution process. The measurements are synthesized to obtain the fault surrogate. The complexity of sequential builds is compared and a new measure, code churn, is calculated. This paper demonstrates the effectiveness of code complexity churn by validating it against the testing problem reports.

In developing a software system, we would like to be able to estimate the way in which the fault content changes during its development, as well as determining the locations having the highest concentration of faults. In the phases prior to test, however, there may be very little direct information regarding the number and location of faults. This lack of direct information requires the development of a fault surrogate from which the number of faults and their location can be estimated. We develop a fault surrogate based on changes in relative complexity, a synthetic measure which has been successfully used as a fault surrogate in previous work. We show that changes in the relative complexity can be used to estimate the rates at which faults are inserted into a system between successive revisions. These rates can be used to continuously monitor the total number of faults inserted into a system, the residual fault content, and identify those portions of a system requiring the application of additional fault detection and removal resources.

This paper presents a new method for measuring the complexity of a software test. This provides a quantitative assessment of each test outcome. Testing effort may be focused on those functions that result in high fractional complexity. The complexity of a test provides a surrogate measure of software faults. The measurement technique is applied to an actual test of a software system.

Software developers use complexity metrics to predict development costs before embarking on a project and to estimate the likelihood of faults once the system is built. Traditional measures, however, were designed principally for sequential programs, providing little insight into the added complexity of concurrent systems or increased demands of real-time systems. For the purpose of predicting cost and effort of development, the CoCoMo model considers factors such as real-time and other performance requirements; for fault prediction, however, most complexity metrics are silent on concurrency. An outline for developing a measure of what we term temporal complexity, including significant and encouraging results of preliminary validation, is presented. 13 standard measures of software complexity are shown to define only two distinct domains of variance in module characteristics. Two new domains of variance are uncovered through 6 out of 10 proposed measures of temporal complexity. The new domains are shown to have predictive value in the modeling of software faults.>

By modeling the relationship between software complexity attributes and software quality attributes, software engineers can take actions early in the development cycle to control the cost of the maintenance phase. The effectiveness of these model-based actions depends heavily on the predictive quality of the model. An enhanced modeling methodology that shows significant improvements in the predictive quality of regression models developed to predict software changes during maintenance is applied here. The methodology reduces software complexity data to domain metrics by applying principal components analysis. It then isolates clusters of similar program modules by applying cluster analysis to these derived domain metrics. Finally, the methodology develops individual regression models for each cluster. These within-cluster models have better predictive quality than a general model fitted to all of the observations.>

Panel Session at the International Symposium on Software Reliability Engineering 1993, Saturday: 6 November 1993, 0830-1000 and 1030-1200

In this paper we examine the quantitative software attributes that are known to be related to software faults and subsequently software failures. The object is to develop design guidelines based on these quantitative attributes that will assist the evaluation of alternatives in the software design process. In order for the several design alternatives to provide meaningful input to the design process, the software system must first be characterized in terms of a series of operational profiles of its projected use.

The authors discuss the development of a complexity domain model for the HAL/S programming language in which all of the space shuttle software is written. Their research has indicated that some aspects of software complexity in other programming language environments are strongly related to software problems incurred during software maintenance. The goal of the current research effort is to establish the particular relationship between the complexity domain model and measurable aspects of software maintenance. From a maintenance perspective, the specific objective is to identify those software complexity attributes that are most closely related to or highly correlated with measures related to software maintenance. This project is expected to lead to the creation of a large database of space system metrics data, the development of a metrics capture tool, and the identification of leading indicators of software maintenance problems.>

The objective in the construction of models of software quality is to use measures that may be obtained relatively early in the software development life cycle to provide reasonable initial estimates of the quality of an evolving software system. Measures of software quality and software complexity to be used in this modeling process exhibit systematic departures of the normality assumptions of regression modeling. Two new estimation procedures are introduced, and their performances in the modeling of software quality from software complexity in terms of the predictive quality and the quality of fit are compared with those of the more traditional least squares and least absolute value estimation techniques. The two new estimation techniques did produce regression models with better quality of fit and predictive quality when applied to data obtained from two software development projects.>

The use of the statistical technique of discriminant analysis as a tool for the detection of fault-prone programs is explored. A principal-components procedure was employed to reduce simple multicollinear complexity metrics to uncorrelated measures on orthogonal complexity domains. These uncorrelated measures were then used to classify programs into alternate groups, depending on the metric values of the program. The criterion variable for group determination was a quality measure of faults or changes made to the programs. The discriminant analysis was conducted on two distinct data sets from large commercial systems. The basic discriminant model was constructed from deliberately biased data to magnify differences in metric values between the discriminant groups. The technique was successful in classifying programs with a relatively low error rate. While the use of linear regression models has produced models of limited value, this procedure shows great promise for use in the detection of program modules with potential for faults.>

The central theme of the study is the creation of a suitable complexity measure for use in software reliability models. Factor analytic techniques are employed to reduce the dimensionality of the complexity problem space to produce a set of reduced metrics. The reduced metrics are subsequently combined into a single relative complexity measure. Program complexity varies dynamically as a function of inputs to the system. Hence, the notion of relative complexity is extended to a dynamic or functional complexity metric for use in proposed modifications to existing reliability models.>

The relationship between measures of software complexity and programming errors is explored. Four distinct regression models were developed for an experimental set of data to create a predictive model from software complexity metrics to program errors. The lines of code metric, traditionally associated with programming errors in predictive models, was found to be less valuable as a criterion measure in these models than measures of software control complexity. A factor analytic technique used to construct a linear compound of lines of code with control metrics was found to yield models of superior predictive quality.>

Predictive models that incorporate a functional relationship of program error measures with software complexity metrics and metrics based on factor analysis of empirical data are developed. Specific techniques for assessing regression models are presented for analyzing these models. Within the framework of regression analysis, the authors examine two separate means of exploring the connection between complexity and errors. First, the regression models are formed from the raw complexity metrics. Essentially, these models confirm a known relationship between program lines of code and program errors. The second methodology involves the regression of complexity factor measures and measures of errors. These complexity factors are orthogonal measures of complexity from an underlying complexity domain model. From this more global perspective, it is believed that there is a relationship between program errors and complexity domains of program structure and size (volume). Further, the strength of this relationship suggests that predictive models are indeed possible for the determination of program errors from these orthogonal complexity domains.>

Software complexity metrics attempt to define the unique characteristics of computer programs in an analytical way.Many such metrics have been developed to explain various perceived differences among programs.Many studies have been conducted to show the similarity among classes of these metrics.What is lacking in this body of literature is a technique which will aid in the establishment of the true dimensionality of the complexity problem space.The objective of this paper is to examine some recent investigations in the area of software complexity using factor analysis to begin an exploration of the actual dimensionality of the complexity metrics.This technique can expose the relationships of these many metrics, one to another.Some correlation coefficients from recent empirical studies on software metrics were factor analyzed, showing the probable existence of five complexity dimensions within thirty five different complexity measures.