Problem reports at NASA are similar to bug reports: they capture defects found during test, post-launch operational anomalies, and document the investigation and corrective action of the issue. These artifacts are a rich source of lessons learned for NASA, but are expensive to analyze since problem reports are comprised primarily of natural language text. We apply topic modeling to a corpus of NASA problem reports to extract trends in testing and operational failures. We collected 16,669 problem reports from six NASA space flight missions and applied Latent Dirichlet Allocation topic modeling to the document corpus. We analyze the most popular topics within and across missions, and how popular topics changed over the lifetime of a mission. We find that hardware material and flight software issues are common during the integration and testing phase, while ground station software and equipment issues are more common during the operations phase. We identify a number of challenges in topic modeling for trend analysis: 1) that the process of selecting the topic modeling parameters lacks definitive guidance, 2) defining semantically-meaningful topic labels requires non-trivial effort and domain expertise, 3) topic models derived from the combined corpus of the six missions were biased toward the larger missions, and 4) topics must be semantically distinct as well as cohesive to be useful. Nonetheless, topic modeling can identify problem themes within missions and across mission lifetimes, providing useful feedback to engineers and project managers.

COUFLESS is a one-day workshop that starts with keynote speaker, Prof. Kishor S. Trivedi from Duke University, North Carolina, USA whose talk's title is titled: "Why Does Software Fail and What Should be Done About It?" A total of 15 papers were submitted to COUFLESS with 53 authors from nine countries and each paper received at least three reviews by the 26 members of the program committee from 11 countries, making it a truly International Workshop. After a long discussion, 11 papers were accepted with the acceptance rate of 73%. Accepted papers address the issues of localizing and debugging complex faults in large-scale software applications.

Faults in software systems can have different characteristics. In an earlier paper, the anomaly reports for a number of JPL/NASA missions were analyzed and the underlying faults were classified as Bohrbugs, non-aging-related Mandelbugs, and aging-related bugs. In another paper the times to failure for each of these fault types were examined to identify trends within missions as well as across the missions. The results of those papers are now starting to provide guidance to improve the dependability of space mission software. Just as there are different types of faults, there are different kinds of mitigations of faults and failures. This paper analyzes the mitigations associated with each fault studied in our previous papers. We identify trends of mitigation type proportions within missions as well as from mission to mission. We also look for relationships between fault types and mitigation types. The results will be used to increase the reliability of space mission software.

The growing complexity of mission-critical space mission software makes it prone to suffer failures during operations. The success of space missions depends on the ability of the systems to deal with software failures, or to avoid them in the first place. In order to develop more effective mitigation techniques, it is necessary to understand the nature of the failures and the underlying software faults. Based on their characteristics, software faults can be classified into Bohrbugs, non-aging-related Mandelbugs, and aging-related bugs. Each type of fault requires different kinds of mitigation techniques. While Bohrbugs are usually easy to fix during development or testing, this is not the case for non-aging-related Mandelbugs and aging-related bugs due to their inherent complexity. Systems need mechanisms like software restart, software replication or software rejuvenation to deal with failures caused by these faults during the operational phase. In a previous study, we classified space mission flight software faults into the three above-mentioned categories based on problems reported during operations. That study concentrated on the percentages of the faults of each type and the variation of these percentages within and across different missions. This paper extends that work by exploring the nature of the times to software failure due to Bohrbugs and non-aging-related Mandelbugs for eight JPL/NASA missions. We start by applying trend tests to the times to failure to check if there is any reliability growth (or decay) for each type of failure. For those times to failure sequences with no trend, we fit distributions to the data sets and carry out goodness-of-fit tests. The results will be used to guide the development of improved operational failure mitigation techniques, thereby increasing the reliability of space mission software.

Often repositories of systems engineering artifacts at NASA's Jet Propulsion Laboratory (JPL) are so large and poorly structured that they have outgrown our capability to effectively manually process their contents to extract useful information. Sophisticated text mining methods and tools seem a quick, low-effort approach to automating our limited manual efforts. Our experiences of exploring such methods mainly in three areas including historical risk analysis, defect identification based on requirements analysis, and over-time analysis of system anomalies at JPL, have shown that obtaining useful results requires substantial unanticipated efforts - from preprocessing the data to transforming the output for practical applications. We have not observed any quick 'wins' or realized benefit from short-term effort avoidance through automation in this area. Surprisingly we have realized a number of unexpected long-term benefits from the process of applying text mining to our repositories. This paper elaborates some of these benefits and our important lessons learned from the process of preparing and applying text mining to large unstructured system artifacts at JPL aiming to benefit future TM applications in similar problem domains and also in hope for being extended to broader areas of applications.

As space mission software becomes more complex, the ability to effectively deal with faults is increasingly important. The strategies that can be employed for fighting a software bug depend on its fault type. Bohrbugs are easily isolated and removed during software testing. Mandelbugs appear to behave chaotically. While it is more difficult to detect these faults during testing, it may not be necessary to correct them; a simple retry after a failure occurrence may work. Aging-related bugs, a sub-class of Mandelbugs, can cause an increasing failure rate. For these faults, proactive techniques may prevent future failures. In this paper, we analyze the faults discovered in the on-board software for 18 JPL/NASA space missions. We present the proportions of the various fault types and study how they have evolved over time. Moreover, we examine whether or not the fault type and attributes such as the failure effect are independent.

Analyzing requirements for consistency and checking them for correctness can require significant effort, particularly if they have not been maintained with a requirements management tool (e.g., DOORS) or specified in a machine-readable notation. By restricting the number of requirements being analyzed, fewer opportunities exist for introducing errors into the analysis. This can be accomplished by subsetting the requirements and analyzing one subset at a time.Previous work showed that simple natural language processing and machine learning techniques can be used to identify temporal requirements within a set of natural language requirements. This paper builds on that work by detailing our results in applying these techniques to a set of natural-language temporal requirements taken from a current JPL mission and determining whether a requirement is one of the most frequently occurring types of temporal requirements.The ability to distinguish between different LTL patterns in natural-language requirements raises the possibility of automating the transformation of natural-language temporal requirements into LTL expressions. This would allow automated consistency checking and tracing of natural-language temporal requirements. Since correctness properties are often specified as LTL expressions, this would also provide a set of correctness properties against which abstract models of the system could be verified.

Over the past several years, we have been developing software fault predictors based on a system's measured structural evolution. We have previously shown significant linear relationships between code churn, a set of synthesized metrics, and the rate at which faults are inserted into the system in terms of number of faults per unit change in code churn. A limiting factor in this and other such investigations has been the absence of a quantitative, consistent and repeatable definition of what constitutes a fault. The rules for fault definition were not sufficiently rigorous to provide unambiguous, repeatable fault counts. Within the framework of a space mission software development effort at the Jet Propulsion Laboratory we have developed a standard for the precise enumeration of faults. This new standard permits software faults to be measured directly from configuration control documents. We compared the new method of counting faults with two existing techniques to determine whether the fault-counting technique has an effect on the quality of the fault models constructed from those counts. The new fault definition provides higher quality fault models than those obtained using the other definitions of fault. Copyright © 2006 John Wiley & Sons, Ltd.

Requirements play a pivotal role in planning, selection, development, testing and operation of NASA's missions. Starting from mission objectives, requirements are successively decomposed into finer detail and ultimately allocated to individual components. This decomposition process is sometimes referred to as "requirements flowdown" between successive levels. The correctness of this decomposition is obviously critical to mission success: if a higher level requirement is improperly decomposed into lower level requirements, the subsequent development from that stage onwards will be jeopardized. A task to determine how to improve the rigor of requirements flowdown analyses has recently been completed. Since requirements continue to be written in natural language, the task focused on examining only natural language specifications for a JPL space mission system. One particular aspect of flowdown analysis examined was classifying requirements by type -subsetting a collection of requirements in this fashion may significantly reduce the effort required to analyze a flowdown by restricting the number of requirements analyzed at any one time, and may increase the accuracy of the flowdown analysis. The application of natural language processing techniques and machine learning techniques has resulted in classifiers that perform well in distinguishing between temporal and non-temporal requirements written in natural language. Such a capability could reduce the effort required to analyze requirements of a given type. Current plans call for extending this capability to improving automated tracing of related requirements and checking a set of related requirements for consistency

Abstract Our current work involves developing methods of measuring changes to evolving software systems. We study a system's change characteristics over a large number of builds using the distinct sources of variation in the software metrics used to measure the system. We have been collaborating with a flight software technology development effort at the Jet Propulsion Laboratory (JPL) and have progressed in resolving the limitations of our earlier work in two distinct steps. First, we have developed a repeatable and consistent fault enumeration methodology, allowing them to be precisely and accurately measured. Second, we have developed a practical framework for automating fault measurement, which we applied to the JPL software system during its development. Every change to the system was measured and every identified fault was tracked to a specific code module. Our analysis indicates that measures of the evolving system's structure are strongly related to the number of faults inserted during its development, and that some types of change are more likely to result in the insertion of faults than others. The fault enumeration methodology ensures that the resulting fault model has greater predictive validity; it also provides a higher quality model than other popular definitions of a fault. Copyright © 2005 John Wiley & Sons, Ltd.

Over the past few years, we have been developing software fault predictors based on a system's measured structural evolution. We have previously shown there is a significant linear relationship between code chum, a set of synthesized metrics, and the rate at which faults are inserted into the system in terms of number of faults per unit change in code chum. A limiting factor in this and other investigations of a similar nature has been the absence of a quantitative, consistent, and repeatable definition of what constitutes a fault. The rules for fault definition were not sufficiently rigorous to provide unambiguous, repeatable fault counts. Within the framework of a space mission software development effort at the Jet Propulsion Laboratory (JPL) we have developed a standard for the precise enumeration of faults. This new standard permits software faults to be measured directly from configuration control documents. Our results indicate that reasonable predictors of the number of faults inserted into a software system can be developed from measures of the system's structural evolution. We compared the new method of counting faults with two existing techniques to determine whether the fault counting technique has an effect on the quality of the fault models constructed from those counts. The new fault definition provides higher quality fault models than those obtained using the other definitions of fault

A critical issue in software engineering is how to construct high assurance software systems, i.e., software Systems where compelling evidence is required that the system delivers its services in a manner satisfying critical properties, such as safety and security. This two-day XSE workshop, the third in a series of workshops on high assurance systems, will provide a forum for researchers and practitioners to exchange ideas and experiences relevant to the development of software for aerospace systems, medical systems, systems controlling nuclear power plants, and other critical systems. Participants of the SEHAS 2003 workshop will explore the opportunities for, and benefits of, synergies between three important themes-product, process, and profiling-each theme reflecting an important aspect of software development for high assurance systems.

Over the past several years, we have been developing methods of measuring the change characteristics of evolving software systems. Not all changes to software systems are equal. Some changes to these systems are very small and have low impact on the system as a whole. Other changes are substantial and have a very large impact of the fault proneness of the complete system. In this study we will identify the sources of variation in the set of software metrics used to measure the system. We will then study the change characteristics to the system over a large number of builds. We have begun a new investigation in these areas in collaboration with a flight software technology development effort at the Jet Propulsion Laboratory (JPL) and have progressed in resolving the limitations of the earlier work in two distinct steps. First, we have developed a standard for the enumeration of faults. This new standard permits software faults to be measured precisely and accurately. Second, we have developed a practical framework for automating the measurement of these faults. This new standard and fault measurement process was then applied to a software system's structural evolution during its development. Every change to the software system was measured and every fault was identified and tracked to a specific code module. The measurement process was implemented in a network appliance, minimizing the impact of measurement activities on development efforts and enabling the comparison of measurements across multiple development efforts. In this paper, we analyze the measurements of structural evolution and fault counts obtained from the JPL flight software technology development effort. Our results indicate that the measures of structural attributes of the evolving software system are suitable for forming predictors of the number of faults inserted into software modules during their development, and that some types of change are more likely to result in the insertion of faults than others. The new fault standard also insures that the model so developed has greater predictive validity.

The NASA Remote Exploration and Experimentation (REE) Project, managed by the Jet Propulsion Laboratory, has the vision of bringing commercial supercomputing technology into space, in a form which meets the demanding environmental requirements, to enable a new class of science investigation and discovery. Dependability goals of the REE system are 99% reliability over 5 years and 99% availability. In this paper we focus on the reliability/availability modeling and analysis of the REE system. We carry out this task using fault trees, reliability block diagrams, stochastic reward nets and hierarchical models. Our analysis helps to determine the ranges of parameters for which the REE dependability goal will be met. The analysis also allows us to assess different hardware and software fault-tolerance techniques.

With the robotic exploration of the Solar System well underway, a logical next step is the development of spacecraft capable of exploring the regions beyond the orbit of Pluto (e.g., the Kuiper Belt and the Oort Cloud), or capable of exploring in situ the nearer extra-solar planetary systems. Barring unforeseen advances in propulsion and communications technologies, the mission times for such voyages will be measured in decades or centuries, and the spacecraft will be required to achieve a much higher degree of autonomous operation than has been the case for previous and current planetary missions. These characteristics pose significant maintenance challenges that must be overcome if such systems are to be successfully fielded. Recent work at the Jet Propulsion Laboratory (JPL) has identified technologies that will be required to develop these types of spacecraft [ChaoOO]. The list below, taken from [ChaoOO], summarizes some of the advances that will be required. 1) Avionics System a) Polymorphic architecture with generic logic blocks b) Self-repaidself-healing techniques

An important aspect of developing models relating the number and type of faults in a software system to a set of structural measurement is defining what constitutes a fault. By definition, a fault is a structural imperfection in a software system that may lead to the system's eventually failing. A measurable and precise definition of what faults are makes it possible to accurately identify and count them, which in turn allows the formulation of models relating fault counts and types to other measurable attributes of a software system. Unfortunately, the most widely-used definitions are not measurable; there is no guarantee that two different individuals looking at the same set of failure reports and the same set of fault definitions will count the same number of underlying faults. The incomplete and ambiguous nature of current fault definitions adds a noise component to the inputs used in modeling fault content. If this noise component is sufficiently large, any attempt to develop a fault model will produce invalid results. As part of our on-going work in modeling software faults, we have developed a method of unambiguously identifying and counting faults. Specifically, we base our recognition and enumeration of software faults on the grammar of the language of the software system. By tokenizing the differences between a version of the system exhibiting a particular failure behavior, and the version in which changes were made to eliminate that behavior, we are able to unambiguously count the number of faults associated with that failure. With modern configuration management tools, the identification and counting of software faults can be automated.

Over the past several years, techniques for estimating software fault content based on measurements of a system's structural evolution during its implementation have been developed. Proper application of the techniques will yield a detailed map of the faults that have been inserted into the system. This information can be used by development organizations to better control the number of residual faults in the operational system. There are several issues that must be resolved if these techniques are to be successfully inserted into a development effort. These issues are identified, as are possibilities for their resolution.

ISSRE99 Panel Proposal

We develop a new metric, relative critical value deviation (RCVD), for classifying and predicting software quality. The RCVD is based on the concept that the extent to which a metric's value deviates from its critical value, normalized by the scale of the metric, indicates the degree to which the item being measured does not conform to a specified norm. For example, the deviation in body temperature above 98.6 Fahrenheit degrees is a surrogate for fever. Similarly, the RCVD is a surrogate for the extent to which the quality of software deviates from acceptable norms (e.g., zero discrepancy reports). Early in development, surrogate metrics are needed to make predictions of quality before quality data are available. The RCVD can be computed for a single metric or multiple metrics. Its application is in assessing newly developed modules by their quality in the absence of quality data. The RCVD is a part of the larger framework of our measurement models that include the use of Boolean discriminant functions for classifying software quality. We demonstrate our concepts using Space Shuttle flight software data.

In developing a software system, we would like to be able to estimate the way in which the fault content changes during its development, as well as determining the locations having the highest concentration of faults. In the phases prior to test, however, there may be very little direct information regarding the number and location of faults. This lack of direct information requires the development of a fault surrogate from which the number of faults and their location can be estimated. We develop a fault surrogate based on changes in relative complexity, a synthetic measure which has been successfully used as a fault surrogate in previous work. We show that changes in the relative complexity can be used to estimate the rates at which faults are inserted into a system between successive revisions. These rates can be used to continuously monitor the total number of faults inserted into a system, the residual fault content, and identify those portions of a system requiring the application of additional fault detection and removal resources.

Most reported experience with software reliability models is from a project's testing phases, during which researchers have little control over the failure data. Since failure data can be noisy and distorted, reported procedures for determining model applicability may be incomplete. To gain additional insight into this problem, we generated forty sets of data by drawing samples from two distributions, which were used as inputs to six different software reliability models. We used several different methods to analyze the applicability of the models. We expected that a model would perform best on the data sets created to comply with the model's assumptions, but initially found that this was not always the case. More detailed examination showed that a model using a data set created to satisfy its assumptions tended to have better prequential likelihood bias, and bias trend measures, although the Kolmogorov-Smirnov test might not be a reliable indicator of the best model. These results indicate that more than one measure should be used to determine model applicability, and that for greater accuracy they be evaluated in sequence rather than simultaneously.

A general combination approach is proposed to address the predictive accuracy problem for software reliability modeling. Instead of relying on any single model, the authors apply several linear combinations of existing software reliability models, called component models, to form a series of linear-combination models. In an experimental investigation with industrial project data, this set of linear combination models was judged to perform better than the component models. For the purpose of automating the procedures in applying existing software reliability models and the linear-combination models, the authors further suggest a computer-aided software engineering tool, called CASRE, to measure software reliability systematically. A discussion is presented of the ideas, approaches, and experimental results of the combination models, as well as the high-level design, structure, and functionality of the CASRE tool.>

A heuristic approach is given to addressing the software reliability modeling problem. The heuristic approach is based on a linear combination of three popular software reliability models. A simple, predetermined combination is suggested by assigning equal weights to each component model for the final delivery of the software reliability prediction. In a preliminary examination, this equally-weighted linear combination (ELC) model is judged to perform well when applied to three published software failure data sets. The authors further present five other sets of software failure data taken from major projects at the Jet Propulsion Laboratory, and apply the ELC model as well as six other popular models for a detailed comparison and evaluation. The evaluation results indicate that the proposed ELC Model not only performs better than all the other models by a wide margin, but also enjoys favorable properties that practitioners would like to see in their software reliability modeling practices. These properties include: simplicity, low-risk, ease of application, and insensitivity to data noise.>