The Annual Technical Report this year is focused on infrastructure reliability. Infrastructure constitutes those things that are apparent only in their absence. We take the infrastructure for granted, assuming it will always be there. We turn on our water facet, and drinkable water has always flowed out, for most of us, most of the time. Our infrastructure is subject to environment breakages (e.g., earthquakes), accidents (e.g., dig ups of cables), sabotage, intrusion, and compromise. Also everyday component, software or system failures can bring our infrastructure down. Our global connectivity and communications, as well as our world wide distributed development and maintenance systems, increase our productivity and efficiency, but can also increase our vulnerabilities. Our critical infrastructures can be found in many places.

A model of software complexity and reliability is developed. It uses an evolutionary process to transition from one software system to the next, while complexity metrics are used to predict the reliability for each system. Our approach is experimental, using data pertinent to the NASA satellite systems application environment. We do not use sophisticated mathematical models that may have little relevance for the application environment. Rather, we tailor our approach to the software characteristics of the software to yield important defect-related predictors of quality. Systems are tested until the software passes defect presence criteria and is released. Testing criteria are based on defect count, defect density, and testing efficiency predictions exceeding specified thresholds. In addition, another type of testing efficiency - a directed graph representing the complexity of the software and defects embedded in the code - is used to evaluate the efficiency of defect detection in NASA satellite system software. Complexity metrics were found to be good predictors of defects and testing efficiency in this evolutionary process.

Abstract The activities of software testing and reliability are integrated for the purpose of demonstrating how the two activities interact in achieving testing efficiency and the reliability resulting from these tests. Integrating means modeling the execution of a variety of tests on a directed graph representation of an example program. A complexity metric is used to construct the nodes, edges, and paths of the example program. Models are developed to represent the efficiency and achieved reliability of black box and white box tests. Evaluations are made of path, independent path, node, program construct, and random tests to ascertain which, if any, is superior with respect to efficiency and reliability. Overall, path testing has the edge in test efficiency. The results depend on the nature of the directed graph in relation to the type of test. Although there is no dominant method, in most cases the tests that provide detailed coverage are better. For example, path testing discovers more faults than independent path testing. Predictions are made of the reliability and fault correction that results from implementing various test strategies. It is believed that these methods can be used by researchers and practitioners to evaluate the efficiency and reliability of other programs. Copyright © 2008 John Wiley & Sons, Ltd.

A software reliability model is used to predict outliers, that is, values that significantly depart from the expected or mean values. In contrast with most projects that cleanse outliers from their databases because their presence distorts results, we tend to like outliers because by predicting them, we can help ensure safety in critical systems. We use the Space Shuttle failure data to make predictions of whether reliability goals are met if outliers should occur during test or the operational mission. In addition, prospective Shuttle software releases are analyzed to detect possible anomalous behavior that would call for re-inspection of the software to bring it into conformance with reliability specifications.

Recent experience and feedback from panels indicates that what the audience likes best is the chance to ask questions, particularly regarding things that might solve problems on their development projects or in their research studies. This panel has been held at every ISSRE since 1997, and has been highly popular. Originally the panel was chaired by John Musa, but Mike Hinchey took over that role last year. The panel has no presentations, only questions. The questions can be on anything, ranging from theory to details of application. The panelists have been selected to make available wide and extensive experience in the field.

We were curious about how well various reliability and testing models would compare with respect to prediction accuracy and testing effectiveness. Therefore, we conducted several experiments to evaluate these properties for the following models: fault tree analysis, geometric and binomial statistical models; and reliability growth models: Yamada S Shape Model, Schneidewind Single Parameter Model, and Schneidewind Software Reliability Model. We developed modified versions of the geometric and binomial models that comprise a new contribution to the body of software reliability research. The Yamada model provided the best prediction accuracy for one of the Shuttle's failure data sets. Future research would involve evaluating all models against multiple data sets.

Software reliability measurement and prediction are used to evaluate model parameters in advance of applying a model. Measurement involves collecting and analyzing data about the observed reliability of software, from which the parameters are estimated, for example, the occurrence of failures during test. Prediction is using a model to forecast future software reliability, for example, time to next failure during operation. In order to demonstrate the prediction methodology, we must use a software reliability model. Since the Schneidewind model has been used on the NASA Shuttle flight software for reliability predictions, and we have a considerable amount of Shuttle failure data, we use the model and data to demonstrate our methodology.

Abstract We experimented with modifying the existing object‐oriented (OO) design and C++ code of a software reliability model. Our purpose was to assess the efficacy of OO methods for performing maintenance on mathematical software, using a real‐world system (NASA Space Shuttle flight software) to illustrate the approach. In this process, we used variants of UML diagrams to modify our design. We found that although a top‐down approach to software maintenance is normally a good idea, it was still necessary to modify the design once the realities of what could be accomplished in the C++ code came to light. As reliability and maintenance are intimately related, we developed reliability risk analysis to show how maintenance changes to our design and code could be used to measure risk. Another maintenance enhancement to the design and code is the use of reliability parameter analysis to assess, in the advance of prediction, the reliability of a set of software releases. We believe this is the first evaluation of software maintenance using OO methods. Copyright © 2007 John Wiley & Sons, Ltd.

In previous research, we showed that risk factors have a significant negative effect on reliability (e.g., failure occurrence). In this research, we show that it is feasible to predict risk (i.e., the probability that risk factors are related to discrepancy reports occurring on a software release). This is an important advance over the previous research because discrepancy reports are available in the requirements phase - when the cost and labor required to correct faults is low, whereas failure data only becomes available in the test phase - when the cost and labor required to correct faults is high. Although using historical failure data to drive traditional software reliability models would produce greater prediction accuracy, the opportunity to provide early prediction of reliability, using risk factors, outweighs this advantage

In general, software reliability models have focused on modeling and predicting the failure detection process and have not given equal priority to modeling the fault correction process. However, it is important to address the fault correction process in order to identify the need for process improvements. Process improvements, in turn, will contribute to achieving software reliability goals. We introduce the concept of a fault correction profile " a set of functions that predict fault correction events as a function of failure detection events. The fault correction profile identifies the need for process improvements and provides information for developing fault correction strategies. Related to the fault correction profile is the goal fault correction profile. This profile represents the fault correction goal against which the achieved fault correction profile can be compared. This comparison motivates the concept of fault correction process instability, and the attributes of instability. Applying these concepts to the NASA Goddard Space Flight Center fault correction process and its data, we demonstrate that the need for process improvement can be identified, and that improvements in process would contribute to meeting product reliability goals.

In general, software reliability models have focused on modeling and predicting the failure detection process and have not given equal priority to modeling the fault correction process. However, it is important to address the fault correction process in order to identify the need for process improvements. Process improvements, in turn, will contribute to achieving software reliability goals. We introduce the concept of a fault correction profile - a set of functions that predict fault correction events as a function of failure detection events. The fault correction profile identifies the need for process improvements and provides information for developing fault correction strategies. Related to the fault correction profile is the goal fault correction profile. This profile represents the fault correction goal against which the achieved fault correction profile can be compared. This comparison motivates the concept of fault correction process instability, and the attributes of instability. Applying these concepts to the NASA Goddard Space Flight Center fault correction process and its data, we demonstrate that the need for process improvement can be identified, and that improvements in process would contribute to meeting product reliability goals.

In general, software reliability models have focused an modeling and predicting failure occurrence and have not given equal priority to modeling the fault correction process. However, there is a need for fault correction prediction, because there are important applications that fault correction modeling and prediction support. These are the following: predicting whether reliability goals have been achieved, developing stopping rules for testing, formulating test strategies, and rationally allocating test resources. Because these factors are related, we integrate them in our model. Our modeling approach involves relating fault correction to failure prediction, with a time delay estimated from a fault correction queuing model.

This panel will discuss the unique challenges and problems of maintaining remote software systems, such as those involved in unmanned planetary missions, satellites, and the Space Shuttle. Little or no software maintenance can be performed on unmanned systems on the ground by humans. Therefore, these systems must have the ability to detect faults and failures automatically and to engage in self-repair with little or no intervention from ground controllers. In the case of planetary missions, the problem is acute due to the great distance separating the vehicle from earth, resulting in signal transmission times so long that a fault in the vehicle could cause the mission to fail, if maintenance depended on human intervention. In addition, planetary probes and satellites must stay operational for years, thus increasing the need for selffailure detection and repair.

In order to continue to make progress in software measurement, as it pertains to reliability and maintainability, we must shift the emphasis from design and code metrics to metrics that characterize the risk of making requirements changes. Although these software attributes can be difficult to deal with due to the fuzzy requirements from which they are derived, the advantage of having early indicators of future software problems outweighs this inconvenience. We developed an approach for identifying requirements change risk factors as predictors of reliability and maintainability problems. Our case example consists of twenty-four Space Shuttle change requests, nineteen risk factors, and the associated failures and software metrics. The approach can be generalized to other domains with numerical results that would vary according to application.

In general, software reliability models have focused on modeling and predicting failure occurrence and have not given equal priority to modeling the fault correction process. However, there is a need for fault correction prediction, because there are important applications that fault correction modeling and prediction support. These are the following: predicting whether reliability goals have been achieved, developing stopping rules for testing, formulating test strategies, and rationally allocating test resources. Because these factors are related, we integrate them in our model. Our modeling approach involves relating fault correction to failure prediction, with a time delay between failure detection and fault correction, represented by a random variable whose distribution parameters are estimated from observed data.

We have developed various software metrics models over the years: Boolean discriminate functions (BDFs); the Kolmogorov-Smirnov distance; derivative calculations for assessing achievable quality; a stopping rule; point and confidence interval estimates of quality; relative critical value deviation metrics; and nonlinear regression functions. We would like these models and metrics to be repeatable across the n builds of a software system. The advantage of repeatability is that models and metrics only need to be developed and validated once on build, and then applied n-1 times without modification to subsequent builds, with considerable savings in analysis and computational effort. In practical terms, this approach involves using the same model parameters that were validated and applying them unchanged on subsequent builds. The disadvantage is that the quality and metrics data of builds 2, ..., n, which varies across builds, is not utilized. We make a comparison of this approach with one that involves validating models and metrics on each build i and applying them only on build i+1, and then repeating the process. The advantage of this approach is that all available data are used in the models and analysis but at considerable cost in effort. We report on experiments involving large sets of discrepancy reports and metrics data on the Space Shuttle flight software, where we compare the predictive accuracy and effort of the two approaches for BDFs, critical values, derivative quality and inspection calculations, and the stopping rule.

We focus on factors that the user should consider when deciding whether to use COTS software. We take the approach of using the common denominator, cost. This is done for two reasons: first, cost is obviously of interest in making such decisions, and second a single metric (cost in dollars), can be used for evaluating the pros and cons of using COTS. The reason is that various software system attributes, like acquisition cost and availability (i.e., the percentage of scheduled operating time that the system is available for use), are commensurate quantities. That is, quantitatively "a low acquisition availability". These units are not multiplicative. However, if it were possible to translate availability into either a cost gain or loss for COTS software, we could operate on these metrics mathematically. Naturally, in addition to cost, the user application is key in making the decision. Thus one could develop a matrix where one dimension is application and the other dimension is the various cost elements. We show how cost elements can be identified and how cost comparisons can be made over the life of the software. Obviously, identifying the costs would not be easy. The user would have to do a lot of work to set up the decision matrix but once it was constructed, it would be a significant tool in the evaluation of COTS. Furthermore, even if all the required data cannot be collected, having a framework that defines software system attributes would serve as a user guide for factors to consider when making the decision about whether to use COTS software or in-house developed software.

We develop a quality control and prediction model for improving the quality of software delivered by development to maintenance. This model identifies modules that require priority attention during development and maintenance. The model also predicts during development the quality that will be delivered to maintenance. We show that it is important to perform a marginal analysis when making a decision about how many metrics to include in a discriminant function. If many metrics are added at once, the contribution of individual metrics is obscured. Also, the marginal analysis provides an effective rule for deciding when to stop adding metrics. We also show that certain metrics are dominant in their effects on classifying quality and that additional metrics are not needed to increase the accuracy of classification. Data from the Space Shuttle flight software are used to illustrate the model process.

We develop a new metric, relative critical value deviation (RCVD), for classifying and predicting software quality. The RCVD is based on the concept that the extent to which a metric's value deviates from its critical value, normalized by the scale of the metric, indicates the degree to which the item being measured does not conform to a specified norm. For example, the deviation in body temperature above 98.6 Fahrenheit degrees is a surrogate for fever. Similarly, the RCVD is a surrogate for the extent to which the quality of software deviates from acceptable norms (e.g., zero discrepancy reports). Early in development, surrogate metrics are needed to make predictions of quality before quality data are available. The RCVD can be computed for a single metric or multiple metrics. Its application is in assessing newly developed modules by their quality in the absence of quality data. The RCVD is a part of the larger framework of our measurement models that include the use of Boolean discriminant functions for classifying software quality. We demonstrate our concepts using Space Shuttle flight software data.

We suggest that empirical studies of maintenance are difficult to understand unless the context of the study is fully defined. We developed a preliminary ontology to identify a number of factors that influence maintenance. The purpose of the ontology is to identify factors that would affect the results of empirical studies. We present the ontology in the form of a UML model. Using the maintenance factors included in the ontology, we define two common maintenance scenarios and consider the industrial issues associated with them. Copyright © 1999 John Wiley & Sons, Ltd.

In analyzing the stability of a software maintenance process, it is important that it is not treated in isolation from the reliability and risk of deploying the software that result from applying the process. Furthermore, we need to consider the efficiency of the test effort that is a part of the process and a determinate of reliability and risk of deployment. The relationship between product quality and process capability and maturity has been recognized as a major issue in software engineering based on the premise that improvements in the process will lead to higher-quality products. To this end, we have been investigating an important facet of process capability-stability-as defined and evaluated by trend, change and shape metrics, across releases and within a release. Our integration of product and process measurement serves the dual purpose of using metrics to assess and predict reliability and risk and to evaluate process stability. We use the NASA Space Shuttle flight software to illustrate our approach.

Obviously, COTS components are different from custom components with respect to one or more of the following attributes: source, development paradigm, safety, reliability, maintainability, availability, security, and other attributes. However, the important question is whether they should be treated differently when deciding to deploy them for operational use; we suggest the answer is no. We use reliability as an example to justify our answer. In order to demonstrate its reliability, a COTS component must pass the same reliability evaluations as the custom components, otherwise the COTS components will be the weakest link in the chain of components and will be the determinant of software system reliability. The challenge is that there will be less information available for evaluating COTS components than for custom components but this does not mean we should despair and do nothing. Actually, there is a lot we can do even in the absence of documentation on COTS components because the customer will have information about how COTS components are to be used in the larger system. To illustrate our approach, we will consider the reliability, maintainability, and availability (RMA) of COTS components as used in larger systems.

In analyzing the stability of a maintenance process, it is important that it not be treated in isolation from the reliability and risk of deploying the software that result from applying the process. Furthermore, we need to consider the efficiency of the test effort that is a part of the process and a determinate of reliability and risk of deployment. Therefore, we integrated these factors into a unified approach. Our contribution to maintenance is the integration and measurement of these factors so that the influence of maintenance actions and test effort on the reliability of the software and the risk of deploying it can be assessed. We use a safety critical application of national visibility-the NASA Space Shuttle-as an example application of the unified approach

Summary form only given. The Space Shuttle Primary Avionics Software Subsystem (PASS) represents a successful integration of many of the computer industry's most advanced software engineering practices and approaches. Beginning in the late 1970's this software development and maintenance project has evolved one of the world's most mature software processes applying the principles of the highest levels of the Software Engineering Institute's Capability Maturity Model and ISO 9001 Standards. This software process, considered to be a "best practice" by many software industry organizations includes state-of-the-practice software reliability engineering (SRE) methodologies. Life-critical PASS produced by this process is recognized to be among the highest quality and highest reliability software in operation in the world. Using this application, we show how SRE can be applied to: interpret software reliability predictions, support verification and validation of the software, assess the risk of deploying the software, predict the reliability of the software, develop test strategies to bring the software into conformance with reliability specifications, and make reliability decisions regarding deployment of the software.

A model is developed that is used to validate and apply metrics for quality control and quality prediction, with the objective of using metrics as early indicators of software quality problems. Metrics and quality factor data from the Space Shuttle flight software are used as an example. Our approach is to integrate quality control and prediction in a single model and to validate metrics with respect to a quality factor. Boolean discriminant functions (BDFs) were developed for use in the quality control and quality prediction process. BDFs provide good accuracy for classifying low quality software because they include additional information for discriminating quality: critical values. Critical values are threshold values of metrics that are used to either accept or reject modules when the modules are inspected during the quality control process. A series of nonparametric statistical methods is also used in the method presented. It is important to perform a marginal analysis when making a decision about how many metrics to use in the quality control and prediction process. We found that certain metrics are dominant in their effects on classifying quality and that additional metrics are not needed to accurately classify quality. This effect is called dominance. Related to the property of dominance is the property of concordance, which is the degree to which a set of metrics produces the same result in classifying software quality. A high value of concordance implies that additional metrics will not make a significant contribution to accurately classifying quality; hence, these metrics are redundant.

Too often, when doing software reliability modeling and prediction, the assumption is made that the software involves either a single module or a single node. The reality in today's increasing use of multi-node client-server systems is that there are multiple software entities that execute on multiple nodes that must be modeled in a system context, if realistic reliability predictions and assessments are to be made. For example, if there are N/sub c/ clients and N/sub x/ servers in a client-server system, it is not necessarily the case that a software failure in any of the N/sub c/ clients or N/sub x/ servers will cause the system to fail. Thus, if such a system were to be modeled as a single entity, the predicted reliability would be much lower than the true reliability, because the prediction would not account for criticality and redundancy. The first factor accounts for the possibility that the survivability of some clients and servers will be more critical to continued system operation than others, while the second factor accounts for the possibility of using redundant nodes to allow for system recovery should a critical node fail. To address this problem, we must identify which nodes-clients and servers-are critical and which are not critical, as defined by whether these nodes are used for critical or non-critical functions, respectively.

We show how residual faults and failures and time to next failure can be used in combination to assist in assuring the safety of the software in safety critical systems like the NASA Space Shuttle Primary Avionics Software System.

Panel Session at the International Symposium on Software Reliability Engineering 1993, Saturday: 6 November 1993, 0830-1000 and 1030-1200

In the use of software reliability models it is not necessarily the case that all the failure data should be used to estimate model parameters and to predict failures. The reason for this is that old data may not be as representative of the current and future failure process as recent data. Therefore it may be possible to obtain more accurate predictions of future failures by excluding or giving lower weight to the earlier failure counts. While there are techniques that involve using trends in failure data or model predictions for selecting reliability data, one software reliability model that has a built-in method for optimally selecting a subset of the failure data for parameter estimation is the Schneidewind Non-Homogeneous Poisson Process (NHPP) software reliability model. We did not find use of this idea in the other models we surveyed. In order to use the concept of "data aging", there must be criterion for determining the optimal value of the starting failure count interval. Our research has identified the mean square error as a good criterion for selecting the starting interval of the failure data. We apply the criterion to select the optimal starting interval. We show that significantly improved reliability predictions can be obtained by using a subset of the failure data, based on applying the criterion, and using the Space Shuttle On-Board software as an example.

The possibility of obtaining more accurate predictions of future failures by excluding or giving lower weight to the earlier failure counts is suggested. Although data aging techniques such as moving average and exponential smoothing are frequently used in other fields, such as inventory control, the author did not find use of data aging in the various models surveyed. A model that includes the concept of selecting a subset of the failure data is the Schneidewind nonhomogeneous Poisson process (NHPP) software reliability model. In order to use the concept of data aging, there must be a criterion for determining the optimal value of the starting failure count interval. Four criteria for identifying the optimal starting interval for estimating model parameters are evaluated The first two criteria treat the failure count interval index as a parameter by substituting model functions for data vectors and optimizing on functions obtained from maximum likelihood estimation techniques. The third uses weighted least squares to maintain constant variance in the presence of the decreasing failure rate assumed by the model. The fourth criterion is the familiar mean square error. It is shown that significantly improved reliability predictions can be obtained by using a subset of the failure data. The US Space Shuttle on-board software is used as an example.>

The use of software reliability models as an aid to software maintenance, with applications to the space shuttle onboard software, is described. In addition to supporting the maintenance function, the use of reliability models throughout the life of the software supports other functions such as reliability assessment, design feasibility assessment, and management of human and computer resources. Reliability prediction during prototype and during test is discussed.>

Using the author's own metrics validation methodology, he shows how this methodology can be used across dissimilar multiple projects. He shows how to reduce the risk of using metrics on multiple projects. He also shows that the choice of metrics and their values can have a significant effect on the quality of software that is achieved and on the cost and amount of inspection that is incurred on multiple projects. A multi-project example, emphasizing the discriminative power validity criterion in support of the quality control function, is presented. A metrics validation process is defined that integrates quality factors, metrics and quality functions.>

A comprehensive metrics validation methodology is proposed that has six validity criteria, which support the quality functions assessment, control, and prediction, where quality functions are activities conducted by software organizations for the purpose of achieving project quality goals. Six criteria are defined and illustrated: association, consistency, discriminative power, tracking, predictability, and repeatability. The author shows that nonparametric statistical methods such as contingency tables play an important role in evaluating metrics against the validity criteria. Examples emphasizing the discriminative power validity criterion are presented. A metrics validation process is defined that integrates quality factors, metrics, and quality functions.>

Metrics that are collected and validated during development can be used during maintenance to control quality and prioritize maintenance work. Validity criteria are defined mathematically. The approach is based on validating selected metrics against related quality factors during development and using the validated metrics during maintenance to: establish initial quality objectives and quality control criteria and prioritize software components (e.g., module) and allocate resources to maintain them. The author illustrates both a case of passing a validation test (discriminative power) and failing a validation test (tracking).>

The author proposes a comprehensive metrics validation methodology that has six validation criteria, each of which supports certain quality functions. New criteria are defined and illustrated, including consistency, discrimination power, tracking, and repeatability. He shows that certain nonparametric statistical methods like contingency tables play an important role in evaluating metrics against the validity criteria. A detailed example emphasizing the discriminative power validity criterion is presented.>

Procedures are proposed to assist the Navy Management Systems Support Office in performing software maintenance. Hardware and software maintenance are contrasted. The key difference between the two -- the ease which software can be changed -- leads to the need for managing software change. Standardization of software is proposed as the method for managing software change. A model of software maintenance is advanced as the function for standardizing software maintenance. (kr)

A paradigm for the system and software design of distributed systems is presented with application to an actual large-scale computer network involving both local area networks and a wide area network. A number of design principles are offered with particular reference to how they can be applied to the design of distributed systems. The author's major point is an explanation of how to make design decisions about distributed systems in a way which will enhance maintainability and understandability of the software and, at the same time, result in good system performance. The aim is to recognize the implications for software quality of various decisions which must be made in the process of specifying a distributed system.>

A state of software maintenance survey is presented, indicating the incongruity of the simultaneous existence of importance and neglect in this field. An overview is given of selected developments and activities covering the following topics: • The "Maintenance Problem." • Models. • Methods for improving maintenance. • Metrics. • Maintenance information management. • Standards. • Maintenance of existing code. • Surveys.

The history of developing and using a simulation model for the study of software error processes, complexity and structure is traced. Strong and weak points of simulation as they relate to model validity, accuracy and cost of implementation and use are discussed. The simulation model is compared to a similar analytic model. The history of an experiment in soft ware complexity and error analysis is used to show the correspondence between empirical and model results. Empirical methods are contrasted with the use of models in terms of validity, accuracy, generality and cost. An assessment is made of the applicability of the techniques, based on these experiences.

The propensity to make programming errors and the rates of error detection and correction are dependent on program complexity. Knowledge of these relationships can be used to avoid errorprone structures in software design and to devise a testing strategy which is based on anticipated difficulty of error detection and correction. An experiment in software error data collection and analysis was conducted in order to study these relationships under conditions where the error data could be carefully defined and collected. Several complexity measures which can be defined in terms of the directed graph representation of a program, such as cyclomatic number, were analyzed with respect to the following error characteristics: errors found, time between error detections, and error correction time. Signifiant relationships were found between complexity measures and error charateristics. The meaning of directed grph structural properties in terms of the complexity of the programming and testing tasks was examined.

A software design of a micro/mini subnet of a campus computer network is described. Software engineering aspects of this design include: identification of terminal, data management and communications functions which are appropriate for micro/mini implementation; logical and physical placement of micro/mini facilities; central versus autonomous operating system control; number and type of protocol layers. The practicality of distributing the above functions at the micro/mini level in a computer network is assessed.