Several metrics have been proposed in the past to quantify the effectiveness of a test suite; they are usually types of coverage metrics, because it is sensible to quantify the effectiveness of a test suite by the extent to which it exercises (covers) various syntactic or semantic features of a program. Though no coverage metric has emerged as the gold standard of test suite effectiveness, mutation coverage is usually considered as a reliable measure thereof, because the ability of a test suite to detect program mutations can be an indication of its ability to detect faults. In this paper, we aim to challenge the superiority of mutation coverage, by showing empirically that the same test suite can have vastly different values of mutation coverage depending on the mutation operators that are used to generate mutants.

When we quantify the effectiveness of a test suite by its mutation coverage, we are in fact equating test suite effectiveness with fault detection: to the extent that mutations are faithful proxies of actual faults, it is sensible to consider that the effectiveness of a test suite to kill mutants reflects its ability to detect faults. But there is another way to measure the effectiveness of a test suite: by its ability to expose the failures of an incorrect program (or, equivalently, its ability to give us confidence in the correctness of a correct program). The relationship between failures and faults is tenuous at best: a fault is the adjudged or hypothesized cause of a failure. Whereas a failure is an observable, verifiable, certifiable effect, a fault is someone’s hypothesis about the possible cause of the observed effect. The same failure may be attributed to more than one fault or combination of faults. In this paper we raise two questions: is the ability to detect faults the same as the ability to reveal failures? If not, which is the better measure of test suite effectiveness? We do not give definite answers to these questions, but we use empirical data to challenge some assumptions and show why these questions are worth answering.