Abstract In statistical testing, a model is developed to characterize the population of uses of the software, and the model is used to generate a statistically correct sample of all uses of the software. A software ‘usage model’ characterizes the population of intended uses of the software in the intended environment. Statistical testing based on a software usage model ensures that the failures that will occur most frequently in operational use will be found early in the testing cycle. The usage model is based on the software specification. The model can be developed in parallel with the software, thus shortening the elapsed time required to develop the deliver software. Usage modeling has been demonstrated to be an activity that improves the specification, gives an analytical description of the specification, quantifies the testing costs and, with statistical testing, provides a basis from which inferences of software reliability may be made. This paper describes the justification for statistical testing of software using a usage model, describes procedures for developing and using a usage model and discusses several usage modeling issues and recent advances in usage model applications.

Abstract ‘Global’ measures of software quality are generally not used by practitioners because they have not been calibrated for local operating environments. A participative process for defining local software quality has been defined, applied in the field, and evaluated in formal validation experiments. In four field applications of the process, a ‘jury’ of seasoned practitioners ranked a sample of modules from the organization's inventory of code and produced a ‘software quality rule set’. The results of each field application were tested in a formal validation experiment to determine whether the obtained rule set was a valid representation of the organization's sense of quality, i.e. whether the locally‐derived rules could enable non‐organizational programmers to make judgments about quality that were similar to those of the jury. In each experiment, non‐organizational programmers ranked the same set of modules that were ranked by the organization's jury, using either the locally‐derived rules, generic (textbook) rules, placebo (non‐helpful) rules, or no rules. The correlation between the judgments of non‐organizational programmers using local rules and the jury was statistically significant at p < 0–005; all other correlations were significant at p < 0–005. The results support a conclusion that the group process for defining local software quality is reliable in producing valid rule sets.

A prototype software system was developed for the U.S. Naval Underwater Systems Center(NUSC) as a demonstration of the Cleanroom Software Engineering methodology. The Cleanroom method is a team approach to the incremental development of software under statistical quality control. Cleanroom's formal methods of Box Structure specification and design, functional verification, and statistical testing were used by a four-person team to develop the Automated Production Control Documentation(APCODOC) system, a relational database application. As is typical in Cleanroom developments, correctness of design and code were ensured through team reviews. Eighteen errors were found during functional verification of the design, and nineteen errors were found during walkthrough of the 1820 lines of FOXBASE code. The software was not executed by developers prior to independent testing (i.e., there was no debugging). There were no errors in compilation, no failures during statistical certification testing, and the software was certified at the target levels of reliability and confidence. Team members attribute the ultimate error-free compilation and failure-free execution of the software to the rigor of the methodology and the intellectual control afforded by the team approach.