We present Flexible Combinatorial Interaction Testing (F-CIT), which aims to improve the flexibility of combinatorial interaction testing (CIT) by eliminating the necessity of developing specialized constructors for CIT problems that cannot be efficiently and effectively addressed by the existing CIT constructors. F-CIT expresses the entities to be covered and the space of valid test cases, from which the samples are drawn to obtain full coverage, as constraints. Computing an F-CIT object (i.e., a set of test cases obtaining full coverage under a given coverage criterion) then turns into an interesting constraint solving problem, which we callcov-CSP. cov-CSP aims to divide the constraints, each representing an entity to be covered, into a minimum number of satisfiable clusters, such that a solution for a cluster represents a test case and the collection of all the test cases generated (one per cluster) constitutes an F-CIT object, covering each required entity at least once. To solve the cov-CSP problem, thus to compute F-CIT objects, we first present two constructors. One of these constructors attempts to cover as many entities as possible in a cluster before generating a test case, whereas the other constructor generates a test case first and then marks all the entities accommodated by this test case as covered. We then use these constructors to evaluate F-CIT in three studies, each of which addresses a different CIT problem. In the first study, we develop structure-based F-CIT objects to obtain decision coverage-adequate test suites. In the second study, we develop order-based F-CIT objects, which enhance a number of existing order-based coverage criteria by taking the reachability constraints imposed by graph-based models directly into account when computing interaction test suites. In the third study, we develop usage-based F-CIT objects to address the scenarios, in which standard covering arrays are not desirable due to their sizes, by choosing the entities to be covered based on their usage statistics collected from the field. We also carry out user studies to further evaluate F-CIT. The results of these studies suggest that F-CIT is more flexible than the existing CIT approaches.

We present a configurable, hybrid, and parallel covering array constructor, called CHiP. CHiP is parallel in that it utilizes vast amount of parallelism provided by graphics processing units (GPUs). CHiP is hybrid in that it bundles the bests of two construction approaches for computing covering arrays; a metaheuristic search-based approach for efficiently covering a large portion of the required combinations and a constraint satisfaction-based approach for effectively covering the remaining hard-to-cover-by-chance combinations. CHiP is configurable in that a trade-off between covering array sizes and construction times can be made. We have conducted a series of experiments, in which we compared the efficiency and effectiveness of CHiP to those of a number of existing constructors by using both full factorial designs and well-known benchmarks. In these experiments, we report new upper bounds on covering array sizes, demonstrating the effectiveness of CHiP, and the first results for a higher coverage strength, demonstrating the scalability of CHiP.