Test coverage is a promising measure of test effectiveness and development organizations are interested in cost-effective levels of coverage that provide sufficient fault removal with contained testing effort. We have conducted a multiple-case study on two dissimilar industrial software projects to investigate if test coverage reflects test effectiveness and to find the relationship between test effort and the level of test coverage. We find that in both projects the increase in test coverage is associated with decrease in field reported problems when adjusted for the number of prerelease changes. A qualitative investigation revealed several potential explanations, including code complexity, developer experience, the type of functionality, and remote development teams. All these factors were related to the level of coverage and quality, with coverage having an effect even after these adjustments. We also find that the test effort increases exponentially with test coverage, but the reduction in field problems increases linearly with test coverage. This suggests that for most projects the optimal levels of coverage are likely to be well short of 100%.

Software product line engineering (SPLE) takes advantage of the commonalities and predicts variabilities among members of a family to create efficient means of producing those members. Different approaches use different methods for defining the product line and producing members of the product line. We describe and demonstrate through a case study the FAST process for achieving generation of members of the product line. The case study introduces the idea of representing constraints among variabilities as edges in a graph and using different graph walking algorithms to generate members of the product line.

Practical model validation techniques are needed for model driven development (MDD) techniques to succeed. This paper presents an approach to generating inputs to test UML design models that are produced in the detailed design phase of an MDD project. A symbolic execution based approach is used to derive test input constraints from the paths of a variable assignment graph, which integrates information from UML class and sequence diagrams. The constraints are solved using Alloy, a configuration constraint solver, to obtain the test inputs. The results of a pilot study carried out to explore the fault detection capability of the test inputs are reported

For model driven development approaches to succeed, there is a need for model validation techniques. This paper presents an approach to testing designs described by UML class diagrams, interaction diagrams, and activity diagrams. A UML design model under test is transformed into an executable form. Test infrastructure is added to the executable form to carry out tests. During testing, object configurations are created, modified and observed. In this paper, we identify the structural and behavioral characteristics that need to be observed during testing. We describe a prototype tool that (1) transforms UML design models into executable forms with test infrastructure, (2) executes tests, and (3) reports failures.

Case studies can help to validate claims that open source software development produces higher quality software at lower cost than traditional commercial development. One problem inherent in case studies are external validity - we do not know whether or not results from one case study apply to another development project. We gain or lose confidence in case study results when similar case studies are conducted on other projects. This case study of the FreeBSD project, a long-lived open source project, provides further understanding of open source development. The paper details a method for mining repositories and querying project participants to retrieve key process information. The FreeBSD development process is fairly well-defined with proscribed methods for determining developer responsibilities, dealing with enhancements and defects, and managing releases. Compared to the Apache project, FreeBSD uses 1) a smaller set of core developers - developers who control the code base - that implement a smaller percentage of the system, 2) a larger set of top developers to implement 80 percent of the system, and 3) a more well-defined testing process. FreeBSD and Apache have a similar ratio of core developers to people involved in adapting and debugging the system and people who report problems. Both systems have similar defect densities and the developers are also users in both systems.