Michael Fagan’s work on the software inspection process defined a practical approach for improving software quality that was broadly adopted throughout the software industry. This work sparked numerous follow-on efforts aimed at understanding and improving the approach. Finally, the process has flexibly evolved over roughly 50 years, incorporating new languages, tools, techniques, and new understandings of its goals and benefits.

Current supercomputing systems consisting of thousands of nodes cannot meet the demands of emerging high-performance scientific applications. As a result, a new generation of supercomputing systems consisting of hundreds of thousands of nodes is being proposed. However, these systems are likely to experience far more frequent failures than today's systems, and such failures must be tackled effectively. Coordinated checkpointing is a common technique to deal with failures in supercomputers. This paper presents a model of a coordinated checkpointing protocol for large-scale supercomputers, and studies its scalability by considering both the coordination overhead and the effect of failures. Unlike most of the existing checkpointing models, the proposed model takes into account failures during checkpointing and recovery, as well as correlated failures. Stochastic activity networks (SANs) are used to model the system, and the model is simulated to study the scalability, reliability, and performance of the system.

The Pioneers' Symposium provides a forum in which Software Engineering's next generation of researchers, faculty, and practitioners have the opportunity to interact with and learn from some of the field's pioneering thinkers. The invited speakers are individuals whose works are not only seminal, but have demonstrated enduring value to both the research and development communities. Attendees will have the opportunity to understand and discuss the speakers' views on (1) what software engineering professionals should know, (2) thinking critically about the field, and (2) what it takes to do work of lasting value.

We derive estimates of mean time to failure and mean time to recover/repair for both hardware and software in a large wireless telecommunications system, based on six months of manually recorded outage data. The observed failure and recovery distributions are not consistent with simple exponential processes. The data can be described by Weibull or two-stage hyper-exponential distributed processes. The duration distributions for scheduled and unscheduled software outages have very different characteristics. The complex distributions observed may be the composition of simple independent processes which cannot be separated in this data set due to a lack of adequately detailed information or proper characterization of outage causes. In this system we found a coverage of /spl sim/98% for autorecovery from unscheduled software failures with an autorepair fraction of /spl sim/36%.

We have found that, when software is developed by multiple, geographically separated teams, the cost-benefit trade-offs of software inspection change. In particular, this situation can significantly lengthen the inspection interval (calendar time needed to complete an inspection). Our research goal was to find a way to reduce the inspection interval without reducing inspection effectiveness. We believed that Internet technology offered some potential solutions, but we were not sure which technology to use nor what effects it would have on effectiveness. To conduct this research, we drew on the results of several empirical studies we had previously performed. These results clarified the role that meetings and individuals play in inspection effectiveness and interval. We conducted further studies showing that manual inspections without meetings were just as effective as manual inspections with them. On the basis of these and other findings and our understanding of Internet technology, we built an economical and effective tool that reduced the interval without reducing effectiveness. This tool, Hypercode, supports meetingless software inspections with geographically distributed reviewers. HyperCode is a platform-independent tool, developed on top of an Internet browser, that integrates seamlessly into the current development process. By seamless, we mean the tool produces a paper flow that is almost identical to the current inspection process. HyperCode's acceptance by its user community has been excellent. Moreover, we estimate that using HyperCode has reduced the inspection interval by 20 to 25 percent. We believe that, had we focused solely on technology (without considering the information our studies had uncovered), we would have created a more complex, but not necessarily more effective tool. We probably would have supported group meetings, restricted each participant's access to review comments, and supported a wider variety of inspection methods. In other words, the principles derived from our empirical studies dramatically and successfully directed our search for a technological solution.

The paper presents the design and implementation of a dependability framework for a call-processing environment in a digital mobile telephone network controller. The framework contains a data audit subsystem to maintain the structural and semantic integrity of the database and a preemptive control flow checking technique, PECOS, to protect call-processing clients. Evaluation of the dependability-enhanced system is performed (using NFTAPE, a software-implemented error injection environment). The evaluation shows that for control flow errors in the client, the combination of PECOS and data audit eliminates fail-silence violations, reduces the incidence of client crashes, and eliminates client hangs. For database injections, data audit detects 85% of the errors and reduces the incidence of escaped errors. Evaluation of combined use of data and control checking (with error injection targeting the database and the client) shows coverage increase from 35% to 80% and indicates data flow errors as a key reason for error escapes.

For years, it was believed that the value of inspections is in finding and fixing defects early in the development process. Otherwise, the cost to find and fix them later is much higher However in examining code inspection data, we are finding that inspections are beneficial for an additional reason. They make the code easier to understand and change. An analysis of data from a recent code inspection experiment shows that 60% of all issues raised in the code inspections are not problems that could have been uncovered by latter phases of testing or field usage because they have little or nothing to do with the visible execution behavior of the software. Rather they improve the maintainability of the code by making the code conform to coding standards, minimizing redundancies, improving language proficiency, improving safety and portability, and raising the quality of the documentation. We conclude that even if advances in software technology have diminished the value of inspections as a defect detection tool, in most cases, it continues to be of value as a maintenance tool.

An essential characteristic of large-scale software development is parallel development by teams of developers. How this parallel development is structured and supported has a profound effect on both the quality and timeliness of the product. We conduct an observational case study in which we collect and analyze the change and configuration management history of a legacy system to delineate the boundaries of, and to understand the nature of, the problems encountered in parallel development. The results of our studies are (1) that the degree of parallelism is very highhigher than considered by tool builders; (2) there are multiple levels of parallelism, and the data for some important aspects are uniform and consistent for all levels; (3) the tails of the distributions are long, indicating the tail, rather than the mean, must receive serious attention in providing solutions for these problems; and (4) there is a significant correlation between the degree of parallel work on a given component and the number of quality problems it has. Thus, the results of this study are important both for tool builders and for process and project engineers.

Software design patterns package proven solutions to recurring design problems in a form that simplifies reuse. We are seeking empirical evidence whether using design patterns is beneficial. In particular, one may prefer using a design pattern even if the actual design problem is simpler than that solved by the pattern, i.e., if not all of the functionality offered by the pattern is actually required. Our experiment investigates software maintenance scenarios that employ various design patterns and compares designs with patterns to simpler alternatives. The subjects were professional software engineers. In most of our nine maintenance tasks, we found positive effects from using a design pattern: either its inherent additional flexibility was achieved without requiring more maintenance time or maintenance time was reduced compared to the simpler alternative. In a few cases, we found negative effects: the alternative solution was less error-prone or required less maintenance time. Overall, we conclude that, unless there is a clear reason to prefer the simpler solution, it is probably wise to choose the flexibility provided by the design pattern because unexpected new requirements often appear. We identify several questions for future empirical research.

Large scale software products must constantly change in order to adapt to a changing environment. Studies of historic data from legacy software systems have identified three specific causes of this change: adding new features; correcting faults; and restructuring code to accommodate future changes. Our hypothesis is that a textual description field of a change is essential to understanding why that change was performed. Also, we expect that difficulty, size, and interval would vary strongly across different types of changes. To test these hypotheses we have designed a program which automatically classifies maintenance activity based on a textual description of changes. Developer surveys showed that the automatic classification was in agreement with developer opinions. Tests of the classifier on a different product found that size and interval for different types of changes did not vary across two products. We have found strong relationships between the type and size of a change and the time required to carry it out. We also discovered a relatively large amount of perfective changes in the system we examined. From this study we have arrived at several suggestions on how to make version control data useful in diagnosing the state of a software project, without significantly increasing the overhead for the developer using the change management system.

An essential characteristic of large scale software development is parallel development by teams of developers. How this parallel development is structured and supported has a profound effect on both the quality and timeliness of the product. We conduct an observational case study in which me collect and analyze the change and configuration management history of a legacy system to delineate the boundaries of, and to understand the nature of, the problems encountered in parallel development. The results of our studies are: 1) that the degree of parallelism is very high-higher than considered by tool builders; 2) there are multiple levels of parallelism and the data for some important aspects are uniform and consistent for all levels and 3) the tails of the distributions are long, indicating the tail, rather than the mean, must receive serious attention in providing solutions for these problems.

In a previous experiment, we determined how various changes in three structural elements of the software inspection process (team size and the number and sequencing of sessions) altered effectiveness and interval. Our results showed that such changes did not significantly influence the defect detection rate, but that certain combinations of changes dramatically increased the inspection interval. We also observed a large amount of unexplained variance in the data, indicating that other factors must be affecting inspection performance. The nature and extent of these other factors now have to be determined to ensure that they had not biased our earlier results. Also, identifying these other factors might suggest additional ways to improve the efficiency of inspections. Acting on the hypothesis that the “inputs” into the inspection process (reviewers, authors, and code units) were significant sources of variation, we modeled their effects on inspection performance. We found that they were responsible for much more variation in detect detection than was process structure. This leads us to conclude that better defect detection techniques, not better process structures, are the key to improving inspection effectiveness. The combined effects of process inputs and process structure on the inspection interval accounted for only a small percentage of the variance in inspection interval. Therefore, there must be other factors which need to be identified.

Process studies and improvement efforts typically call for new instrumentation on the process in order to collect the data they have deemed necessary. This can be intrusive and expensive, and resistance to the extra workload often foils the study before it begins. The result is neither interesting new knowledge nor an improved process. In many organizations, however, extensive historical process and product data already exist. Can these existing data be used to empirically explore what process factors might be affecting the outcome of the process? If they can, organizations would have a cost-effective method for quantitatively, if not causally, understanding their process and its relationship to the product. We present a case study that analyzes an in-place industrial process and takes advantage of existing data sources. In doing this, we also illustrate and propose a methodology for such exploratory empirical studies. The case study makes use of several readily-available repositories of process data in the industrial organization. Our results show that readily available data can be used to correlate both simple aggregate metrics and complex process metrics with defects in the product. Through the case study, we give evidence supporting the claim that exploratory empirical studies can provide significant results and benefits while being cost-effective in their demands on the organization.

Testing commercial software is expensive and time consuming.Automated testing methods promise to save a great deal of time and money throughout the software industry.One approach that is well-suited for the reactive systems found in telephone switching systems is specification-based testing.We have built a set of tools to automatically test software applications for violations of safety properties expressed in temporal logic.Our testing system automatically constructs finite state machine oracles corresponding to safety properties, builds test harnesses, and integrates them with the application.The test harness then generates inputs automatically to test the application.We describe a study examining the feasibility of this approach for testing industrial applications.To conduct this study we formally modeled an Automatic Protection Switching system (APS), which is an application common to many telephony systems.We then asked a number of computer science graduate students to develop several versions of the APS and use our tools to test them.We found that the tools are very effective, save significant amounts of human effort (at the expense of machine resources), and are easy to use.We also dis-

and the time savings from the reduction in distribution interval of the inspection package (sometimes involving international mailings) have been substantial.These savings together with the seamless integration into the existing environment are the major factors for this acceptance.From our viewpoint as experimentalists, the acceptance came too readily.Therefore we lost our opportunity to explore this tool using a series of controlled experiments to isolate the underlying factors or its effectiveness.Nevertheless, by using historical data we can show that the new process is less expensive in terms of cost and at least as effective in terms of quality (defect detection effectiveness).

No abstract available.

We have conducted an industrial experiment to assess the cost-benefit tradeoffs of several software inspection processes. Our results to date explain the variation in observed effectiveness very well, but are unable to satisfactorily explain variation in inspection interval. In this article we examine the effect of a new factor - process environment - on inspection interval (calendar time needed to complete the inspection). Our analysis suggests that process environment does indeed influence inspection interval. in particular, we found that non-uniform work priorities, time-varying workloads, and deadlines have significant effects. Moreover, these experiences suggest that regression models are inherently inadequate for interval modeling, and that queueing models may be more effective. (Also cross-referenced as UMIACS-TR-97-19)

We conducted a long term experiment to compare the costs and benefits of several different software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development effort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. The article has several goals: (1) to describe the experiment's design and show how we used simulation techniques to optimize it; (2) to present our results and discuss their implications for both software practitioners and researchers; and (3) to discuss several new questions raised by our findings. For each inspection, we randomly assigned three independent variables: (1) the number of reviewers on each inspection team (1, 2, or 4); (2) the number of teams inspecting the code unit (1 or 2); and (3) the requirement that defects be repaired between the first and second team's inspections. The reviewers for each inspection were randomly selected without replacement from a pool of 11 experienced software developers. The dependent variables for each inspection included inspection interval (elapsed time), total effort, and the defect detection rate. Our results showed that these treatments did not significantly influence the defect detection effectiveness, but that certain combinations of changes dramatically increased the inspection interval.

Article Free Access Share on Experimental software engineering: a report on the state of the art Authors: Lawrence G. Votta Software & Systems Research Lab, AT&T Bell Laboratories, Naperville, Illinois Software & Systems Research Lab, AT&T Bell Laboratories, Naperville, IllinoisView Profile , Adam Porter University of Maryland, Computer Science Department, College Park, Maryland University of Maryland, Computer Science Department, College Park, MarylandView Profile Authors Info & Claims ICSE '95: Proceedings of the 17th international conference on Software engineeringApril 1995Pages 277–279https://doi.org/10.1145/225014.225040Published:23 April 1995Publication History 29citation568DownloadsMetricsTotal Citations29Total Downloads568Last 12 Months90Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

We are conducting a long-term experiment (in progress) to compare the costs and benefits of several different software inspection methods.These methods are being applied by beliefs about the most cost-effective ways to conduct inspections and raise some questions about the feasibility of recently proposed methods.professional developers to a commercial software product

Software requirements specifications (SRS) are often validated manually. One such process is inspection, in which several reviewers independently analyze all or part of the specification and search for faults. These faults are then collected at a meeting of the reviewers and author(s). Usually, reviewers use Ad Hoc or Checklist methods to uncover faults. These methods force all reviewers to rely on nonsystematic techniques to search for a wide variety of faults. We hypothesize that a Scenario-based method, in which each reviewer uses different, systematic techniques to search for different, specific classes of faults, will have a significantly higher success rate. We evaluated this hypothesis using a 3/spl times/2/sup 4/ partial factorial, randomized experimental design. Forty eight graduate students in computer science participated in the experiment. They were assembled into sixteen, three-person teams. Each team inspected two SRS using some combination of Ad Hoc, Checklist or Scenario methods. For each inspection we performed four measurements: (1) individual fault detection rate, (2) team fault detection rate, (3) percentage of faults first identified at the collection meeting (meeting gain rate), and (4) percentage of faults first identified by an individual, but never reported at the collection meeting (meeting loss rate). The experimental results are that (1) the Scenario method had a higher fault detection rate than either Ad Hoc or Checklist methods, (2) Scenario reviewers were more effective at detecting the faults their scenarios are designed to uncover, and were no less effective at detecting other faults than both Ad Hoc or Checklist reviewers, (3) Checklist reviewers were no more effective than Ad Hoc reviewers, and (4) Collection meetings produced no net improvement in the fault detection rate-meeting gains were offset by meeting losses.>

Features are often the basic unit of development for a very large software system and represent long-term efforts, spanning up to several years from inception to actual use. Developing an experiment to monitor (by means of sampling) such lengthy processes requires a great deal of care in order to minimize casts and to maximize benefits. Just as prototyping is often a necessary auxiliary step in a large-scale, long-term development effort, so, too, is prototyping a necessary step in the development of a large-scale, long-term process monitoring experiment. Therefore, we have prototyped our experiment using a representative process and reconstructed data from a large and rich feature development. This approach has yielded three interesting sets of results. First, we reconstructed a 30-month time diary for the lead engineer of a feature composed of both hardware and software. These data represent the daily state (where the lead engineer spent the majority of his time) for a complete cycle of the development process. Second, we found that we needed to modify our experimental design. Our initial set of states did not represent the data as well as we had hoped. This is exemplified by the fact that the "Other" category is too large. Finally, the data provide evidence for both a waterfall view and an interactive, cyclic view of software development. We conclude that the prototyping effort is a necessary part of developing and installing any large-scale process monitoring experiment.>

At each step in large software development, reviewers carry out inspections to detect faults. These inspections are usually followed by a meeting to collect the faults that have been discovered. However, we have found that these inspection meetings are not as beneficial as managers and developers think they are. Even worse, they cost much more in terms of products development interval and developer's time than anyone realizes.Analysis of the inspection and collection process leads us to make the following suggestions. First, at the least, the number of participants required at each inspection meeting should be minimized. Second, we propose two alternative fault collection methods, either of which would eliminate the inspection meetings altogether: (a) collect faults by deposition (small face-to-face meetings of two or three persons), or (b) collect faults using verbal or written media (telephone, electronic mail, or notes).We believe that such a change in procedure would increase efficiency by reducing production times without sacrificing product quality.

The number of faults not discovered by the design review can be estimated by using capture-recapture methods. Since these methods were developed for wildlife population estimation, the assumptions used to derive them do not match design review applications. The authors report on a Monte Carlo simulation to study the effects of broken assumptions on maximum likelihood estimators (MLEs) and jackknife estimators (JEs) of faults remaining. It is found that the MLE performs satisfactorily if faults are classified into a small number of homogeneous groups. Without grouping, the MLE can perform poorly, but it generally does better than the JE.>

Article Estimating software fault content before coding Share on Authors: Stephen G. Eick View Profile , Clive R. Loader View Profile , M. David Long View Profile , Lawrence G. Votta View Profile , Scott Vander Wiel View Profile Authors Info & Claims ICSE '92: Proceedings of the 14th international conference on Software engineeringJune 1992 Pages 59–65https://doi.org/10.1145/143062.143090Online:01 June 1992Publication History 83citation658DownloadsMetricsTotal Citations83Total Downloads658Last 12 Months12Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access