ABSTRACT Inspections and testing are two of the most commonly performed software quality assurance processes today. Typically, these processes are applied in isolation, which, however, fails to exploit the benefits of systematically combining and integrating them. In consequence, tests are not focused on the basis of early defect detection data. Expected benefits of such process integration include higher defect detection rates or reduced quality assurance effort. Moreover, when conducting testing without any prior information regarding the system's quality, it is often unclear how to focus testing. A systematic integration of inspection and testing processes requires context‐specific knowledge about the relationships between inspections and testing. This knowledge is typically not available and needs to be empirically identified and validated. Often, context‐specific assumptions can be seen as a starting point for generating such knowledge. On the basis of the integrated inspection and testing approach, In2Test, which uses inspection data to focus testing, we present in this article how knowledge about the relationship between inspections and testing can be gained, documented, and evolved in an analytical or empirical manner. In addition, this article gives an overview of related work and highlights future research directions. Copyright © 2012 John Wiley & Sons, Ltd.

Inspections and testing are two of the most commonly performed software quality assurance processes today. Typically, these processes are applied in isolation, which, however, fails to exploit the benefits of systematically combining and integrating them. Expected benefits of such process integration are higher defect detection rates or reduced quality assurance effort. Moreover, when conducting testing without any prior information regarding the system's quality, it is often unclear which parts or which defect types should be prioritized. Existing approaches do not explicitly use information from inspections in a systematical way to focus testing processes. In this article, we present an integrated two-stage approach that routes inspection data to test processes in order to prioritize code classes and defect types. While an initial version of the approach focused on prioritizing code classes, this article focuses on the prioritization of defect types for testing. Results from a case study where the approach was applied on the code level show that those defect types could be prioritized before the testing that afterwards actually showed up most often during the test process. In addition, an overview of related work and an outlook on future research directions are given.

A systematic approach to decision making in software engineering is required, for instance, if an organization aims at achieving CMMI level three. Rational decision making regarding the selection and introduction of SE technologies requires adequate information about their suitability for the intended organizational context. Research is often unable to provide such information, and this could be one reason why promising techniques are sometimes not adopted in practice. From a research point of view, successful technology transfer requires knowing which information decision makers in industry need, and where they actually look for it. With this knowledge, empirical research can strive to produce the needed information in order to increase the likelihood of successful technology adoption. To address these questions, we conducted an online survey among German software industry decision makers. To focus the survey, we used inspections as an exemplary technology. We invited 9653 companies to participate, from which we received 92 fully completed questionnaires. Our main findings are that information regarding the impact of technologies on product quality, cost, and development time, as well as on technology cost-benefit ratio is considered most important among decision makers. The preferred sources of information are colleagues, textbooks, and industry workshops.

There is a general agreement among software engineering practitioners that software inspections are an important technique to achieve high software quality at a reasonable cost. However, there are many ways to perform such inspections and many factors that affect their cost-effectiveness. It is therefore important to be able to estimate this cost-effectiveness in order to monitor it, improve it, and convince developers and management that the technology and related investments are worth while. This work proposes a rigorous but practical way to do so. In particular, a meaningful model to measure cost-effectiveness is proposed and a method to determine cost-effectiveness by combining project data and expert opinion is described. To demonstrate the feasibility of the proposed approach, the results of a large-scale industrial case study are presented and an initial validation is performed.

Software inspection is one of the most effective methods to detect defects. Reinspection repeats the inspection process for software products that are suspected to contain a significant number of undetected defects after an initial inspection. As a reinspection is often believed to be less efficient than an inspection an important question is whether a reinspection justifies its cost. In this paper we propose a cost-benefit model for inspection and reinspection. We discuss the impact of cost and benefit parameters on the net gain of a reinspection with empirical data from an experiment in which 31 student teams inspected and reinspected a requirements document. Main findings of the experiment are: a) For reinspection benefits and net gain were significantly lower than for the initial inspection. Yet, the reinspection yielded a positive net gain for most teams with conservative cost-benefit assumptions. B) Both the estimated benefits and number of major defects are key factors for reinspection net gain, which emphasizes the need for appropriate estimation techniques.

Explicit risk management is gaining ground in industrial software development projects. However, there are few empirical studies that investigate the transfer of explicit risk management into industry, the adequacy of the risk management approaches to the constraints of industrial contexts, or their cost-benefit. This paper presents results from a case study that introduced a systematic risk management method, namely the Riskit method, into a large German telecommunication company. The objective of the case study was (1) to analyze the usefulness and adequacy of the Riskit method and (2) to analyze the cost-benefit of the Riskit method in this industrial context. The results of (1) also aimed at improvement and customization of the Riskit method. Moreover, we compare our findings with results of previous case studies to obtain more generalized conclusions on the Riskit method. Our results showed that the Riskit method is practical, adds value to the project, and that its key concepts are understood and usable in practice. Additionally, many lessons learned are reported that are useful for the general audience who wants to transfer risk management into new projects.

There is a general agreement among software engineering practitioners that sofware inspections are an important technique to achieve high software quality at a reasonable cost. However, there are many ways to perform such inspections and many factors that affect their cost-effectiveness. It is therefore important to be able to estimate this cost-effectiveness in order to monitor it, improve it, and convince developers and management that the technology and related investments are worthwhile. This work proposes a rigorous but practical way to do so. In particular, a meaningful model to measure cost-effectiveness is selected and a method to determine the cost-effectiveness by combining project data and expert opinion is proposed. To demonstrate the feasibility of the proposed approach, the results of a large-scale industrial case study are presented.

An important requirement to control the inspection of software artifacts is to be able to decide, based on more objective information, whether the inspection can stop or whether it should continue to achieve a suitable level of artifact quality. A prediction of the number of remaining defects in an inspected artifact can be used for decision making. Several studies in software engineering have considered capture-recapture models to make a prediction. However, few studies compare the actual number of remaining defects to the one predicted by a capture-recapture model on real software engineering artifacts. The authors focus on traditional inspections and estimate, based on actual inspections data, the degree of accuracy of relevant state-of-the-art capture-recapture models for which statistical estimators exist. In order to assess their robustness, we look at the impact of the number of inspectors and the number of actual defects on the estimators' accuracy based on actual inspection data. Our results show that models are strongly affected by the number of inspectors, and therefore one must consider this factor before using capture-recapture models. When the number of inspectors is too small, no model is sufficiently accurate and underestimation may be substantial. In addition, some models perform better than others in a large number of conditions and plausible reasons are discussed. Based on our analyses, we recommend using a model taking into account that defects have different probabilities of being detected and the corresponding Jackknife Estimator. Furthermore, we calibrate the prediction models based on their relative error, as previously computed on other inspections. We identified theoretical limitations to this approach which were then confirmed by the data.

In order to control inspections, the number of remaining defects in software artifacts after their inspection should be estimated. This would allow, for example, deciding whether a reinspection of supposedly faulty artifacts is necessary. Several studies in software engineering have considered capture-recapture models for performing such estimations. These models were initially developed for estimating animal abundance in wildlife research. In addition to these models, researchers in software engineering have recently proposed an alternative approach, namely the detection profile method (DPM), that makes less restrictive assumptions than some capture-recapture models and that show promise in terms of estimation accuracy. The authors investigate how to select between these two approaches for defect content estimation. As a result of this investigation they present a selection procedure taking into account the strength and weaknesses of the two methods. A weakness known for capture-recapture models is that they tend to provide extreme under/over estimation. The existence of such extreme outliers can discourage their use because their consequences in terms of wasted effort or defect slippage can be substantial, and therefore it is not clear,whether a particular estimate can be trusted. The evaluation of the selection procedure with actual inspection data indicates that this selection procedure provides the same accuracy as capture-recapture models alone and DPM alone, and most importantly does not exhibit extreme over/under estimation.

An important requirement to control the inspection of software artifacts is to be able to decide, based on objective information, whether inspection can stop or whether it should continue to achieve a suitable level of artifact quality. Several studies in software engineering have considered the use of capture-recapture models to predict the number of remaining defects in an inspected document as a decision criterion about reinspection. However, no study on software engineering artifacts compares the actual number of remaining defects to the one predicted by a capture-recapture model. Simulations have been performed but no definite conclusions can be drawn regarding the degree of accuracy of such models under realistic inspection conditions, and the factors affecting this accuracy. Furthermore, none of these studies performed an exhaustive comparison of existing models. In this study, we focus on traditional inspections and estimate, based on actual inspection data, the degree of accuracy of all relevant, state-of-the-art, capture-recapture models for which statistical estimators exist. We compare the various models' accuracies and look at the impact of the number of inspectors on these accuracies. Results show that model accuracies are strongly affected by the number of inspectors and, therefore, one must consider this factor before using capture-recapture models. When the number of inspectors is below 4, no model is sufficiently accurate and underestimation may be substantial. In addition, some models perform better than others in a large number of conditions and plausible reasons are discussed. Based on our analyses, we recommend using a model taking into account different probabilities of detecting defects and a Jacknife estimator.