Spectrum-based fault localization (SBFL) works well for single-fault programs but its accuracy decays for increasing fault numbers. We present FLITSR (Fault Localization by Iterative Test Suite Reduction), a novel SBFL approach that improves the localization of a given SBFL base metric specifically in the presence of multiple faults. FLITSR iteratively selects reduced versions of the test suite that better localize the individual faults in the system. This allows it to identify and re-rank faults ranked too low by the base metric because they were masked by other program elements. Through this process, FLITSR returns a set of highly suspicious program elements (called a basis), where the execution of each failing test involves at least one basis element, considered as the cause of the failure. We implemented the FLITSR algorithm in an open source toolset and extensively evaluated it over three true multi-fault datasets, varying the fault type, coverage granularity and programming language. Our evaluation shows that FLITSR consistently outperforms existing localization metrics and methods, including those designed to handle multiple faults such as ARTEMIS and parallel debugging. For the Defects4J method-level faults, FLITSR also substantially outperforms GRACE, a state-of-the-art learning-based fault localizer.

This report details the contents of the artifact for the paper “FLITSR: Improved Spectrum-Based Localization of Multiple Faults by Iterative Test Suite Reduction,” as well as detailed instructions for its installation and usage. The artifact contains all the necessary components to replicate the results of the study, including the three datasets used as well as the FLITSR tool which is used as both the implementation of the FLITSR algorithm and the evaluation framework for the experiments in the study.

Datasets such as Defects4J and BugsInPy that contain bugs from real-world software projects are necessary for a realistic evaluation of automated debugging tools. However, these datasets largely identify only a single bug in each entry, while real-world software projects (including those used in Defects4J and BugsInPy) typically contain multiple bugs at the same time. We lift this limitation and describe an automated approach to identify multiple existing bugs in the individual dataset entries. We use test case transplantation and fault location translation to expose and locate the bugs, respectively. We identified 9.2 faults on average in each of the 311 versions of the 5 projects in Defects4J, and 18.6 faults in 501 versions of the 17 projects in BugsInPy. We thus provide datasets of true multi-fault versions within real-world software projects, which maintain the properties, format and usability of the original datasets.

Spectrum-based fault localization (SBFL) works well for single-fault programs but its accuracy decays for increasing fault numbers. We present FLITSR (Fault Localization by Iterative Test Suite Reduction), a novel SBFL extension that improves the localization of a given base metric specifically in the presence of multiple faults. FLITSR iteratively selects reduced versions of the test suite that better localize the individual faults in the system. This allows it to identify and re-rank faults ranked too low by the base metric because they were masked by other program elements.