Abstract Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. attempts to estimate suspiciousness more accurately using logistic regression analysis. attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that has very high accuracy, and can drastically reduce time cost for targets that have been difficult to complete by the conventional method.

Combinatorial testing is a widely used technique to detect failures caused by interactions of system under test (SUT) parameters. Faulty interaction localization (FIL) is a problem to locate parameter-value combinations that trigger failures from combinatorial test cases and their testing results. FIL is important for debugging, but is expensive for large test suites and SUTs since the number of candidates of faulty interactions increases exponentially with the number of parameters and the size of interactions. To address this problem, this paper proposes a method employing logistic regression. The proposed FIL based on Regression coefficients Of loGistic regression analysis (called FROG) computes the suspiciousness of each parameter-value combination to be included in a faulty interaction from its corresponding regression coefficient. We evaluate the proposed method by applying FROG to combinatorial t-way test cases (2 ≤ t ≤ 4) for real application SUT models, e.g. TCAS, GCC, and Apache. Our experiment results show that FROG can effectively locate faulty interactions injected while efficiently reducing the number of candidates of potential faulty interactions to be checked.

This paper proposes a novel approach to combinatorial test generation, which achieves an increase of not only the number of new combinations but also the distance between test cases. We applied our distance-integrated approach to a state-of-the-art greedy algorithm for traditional combinatorial test generation by using two distance metrics, Hamming distance, and a modified chi-square distance. Experimental results using numerous benchmark models show that combinatorial test suites generated by our approach using both distance metrics can improve interaction coverage for higher interaction strengths with low computational overhead.

Combinatorial testing aims at covering the interactions of parameters in a system under test, while some combinations may be forbidden by given constraints (forbidden tuples). In this paper, we illustrate that such forbidden tuples correspond to unsatisfiable cores, a widely understood notion in the SAT solving community. Based on this observation, we propose a technique to detect forbidden tuples lazily during a greedy test case generation, which significantly reduces the number of required SAT solving calls. We further reduce the amount of time spent in SAT solving by essentially ignoring constraints while constructing each test case, but then “amending” it to obtain a test case that satisfies the constraints, again using unsatisfiable cores. Finally, to complement a disturbance due to ignoring constraints, we implement an efficient approximative SAT checking function in the SAT solver Lingeling. Through experiments we verify that our approach significantly improves the efficiency of constraint handling in our greedy combinatorial testing algorithm.

Combinatorial testing is a widely-used technique to detect system interaction failures. To improve test effectiveness with given priority weights of parameter values in a system under test, prioritized combinatorial testing constructs test suites where highly weighted parameter values appear earlier or more frequently. Such order-focused and frequency-focused combinatorial test generation algorithms have been evaluated using metrics called weight coverage and KL divergence but not sufficiently with fault detection effectiveness so far. We evaluate the fault detection effectiveness on a collection of open source utilities, applying prioritized combinatorial test generation and investigating its correlation with weight coverage and KL divergence.

Once a bug in software is reported, developers have to determine which source files are related to the bug. This process is referred to as bug localization, and an automatic way of bug localization is important to improve developers' productivity. This paper proposes an approach called DrewBL to efficiently localize faulty files for a given bug report using a natural language processing tool, word2vec. In DrewBL, we first build a vector space model named semantic-VSM which represents a distributed representation of words in the bug report and source code files and next compute the relevance between them by feeding the constructed model to word2vec. We also present an approach called CombBL to further improve the accuracy of bug localization which employs not only the proposed DrewBL but also existing bug localization techniques, such as BugLocator based on textual similarity and Bugspots based on bug-fixing history, in a combinational manner. This paper gives our early experimental results to show the effectiveness and efficiency of the proposed approaches using two open source projects.

Priorities (weights) for parameter values can improve the effectiveness of combinatorial testing. Previous approaches have employed weights to derive high-priority test cases either earlier or more frequently. Our approach integrates these order-focused and frequency-focused prioritizations. We show that our priority integration realizes a small test suite providing high-priority test cases early and frequently in a good balance. We also propose two algorithms that apply our priority integration to existing combinatorial test generation algorithms. Experimental results using numerous test models show that our approach improves the existing approaches w.r.t. Order-focused and frequency-focused metrics, while overheads in the size and generation time of test suites are small.

Combinatorial testing aims at reducing the cost of software and system testing by reducing the number of test cases to be executed. We propose an approach for combinatorial testing that generates a set of test cases that is as small as possible, using incremental SAT solving. We present several search-space pruning techniques that further improve our approach. Experiments show a significant improvement of our approach over other SAT-based approaches, and considerable reduction of the number of test cases over other combinatorial testing tools.

Modbat makes testing easier by providing a user-friendly modeling language to describe the behavior of systems, from such a model, test cases are generated and executed. Modbat's domain-specific language is based on Scala, its features include probabilistic and non-deterministic transitions, component models with inheritance, and exceptions. We demonstrate the versatility of Modbat by finding a confirmed defect in the currently latest version of Java, and by testing SAT solvers.

In this paper, we develop a combinatorial testing technique for tree-structured test models. First, we generalize our previous test models for combinatorial testing based on and-xor trees with constraints limited to a syntactic subset of propositional logic, to allow for constraints in full propositional logic. We prove that the generalized test models are strictly more expressive than the limited ones. Then we develop an algorithm for combinatorial testing for the generalized models, and show its correctness and computational complexity. We apply a tool based on our algorithm to an actual ticket gate system that is used by several large transportation companies in Japan. Experimental results show that our technique outperforms existing techniques.

In this paper, we quantitatively analyze the efficiency of the Model-Based Review (MBR) method in an actual software design from the two points of view; cost and reviewability. The MBR method is a modeling procedure for the purpose of reviewing preliminary design specifications of web-based applications. We have collected process data in applying both of the MBR method and an ordinary review to a preliminary design of a developing web-based library system. Analyzing the collected process data, we quantitatively compare the efficiency of the MBR method and that of the ordinary review. As a result of this comparative analysis, we show that the MBR method is superior to the ordinary review in terms of not only reviewability but also cost through the experimental design process.

This paper proposes an approach for verifying class specifications of Web applications using model checking. We first present a method to model a dynamic behavior of a Web application from a class specification. We next propose two methods to verify consistencies of the class specification and other design specifications: (1) a page flow diagram which is one of the most essential specifications for Web applications and (2) a behavior diagram such as a UML activity diagram. We applied the proposed methods to real specifications of a Web application designed by a certain company and found several faults of the specifications that had not been detected in actual reviews.