Abstract Combinatorial interaction testing (CIT) aims at constructing a covering array (CA) of all value combinations at a specific interaction strength, to detect faults that are caused by the interaction of parameters. CIT has been widely used in different applications, with many algorithms and tools having been proposed to support CA construction. To date, however, there appears to have been no studies comparing different CA constructors when only some of the CA test cases are executed. In this paper, we present an investigation of five popular CA constructors: ACTS, Jenny, PICT, CASA and TCA. We conducted empirical studies examining the five programs, focusing on interaction coverage and fault detection. The experimental results show that when there is no preference or special justification for using other CA constructors, then Jenny is recommended—because it achieves better interaction coverage and fault detection than the other four constructors in many cases. Our results also show that when using ACTS or CASA, their CAs must be prioritized before testing. The main reason for this is that these CAs can result in considerable interaction coverage or fault detection capabilities when executing a large number of test cases; however, they may also produce the lowest rates of fault detection and interaction coverage.

Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread-applicability to programs with nonnumeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate “distance” measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the failure-detection effectiveness of our technique by performing an empirical study on 14 object programs, using two standard metrics-F-measure and P-measure. Our ART algorithm statistically significantly outperforms RT on 10 of the 14 programs studied, and exhibits performance similar to RT on three of the four remaining programs. The selection overhead of our ART algorithm is close to that of RT.

This paper introduces a testing strategy that is suitable for testing service-based applications. We describe an architecture that responds to changes of service operation, operation arguments and service composition changes. Our proof-of-concept test system performs runtime testing on our model atomic and composite web services using a random testing technique. A novel change identification method was developed to capture changes at the service interface. The test system is able to identify changes that occur in service operations and operational arguments in a service description of a test candidate. Our approach uses a new method to detect changes in a service inventory. Automated reconfiguration is used to support the continuous operation of the testing systems during a test candidate change.

The broad domain of software testing includes different job responsibilities such as creating test plans, devising and running a variety of tests, documenting results, to liaising between different development teams. In this paper, we attempt to collate a list of software testing job responsibilities by applying three different social research methodologies to collect information from different sources. We found that “test” specific responsibilities are divided into several unit tasks including test suite generation, execution of test plans, and so on. We also found that along with test specific responsibilities, software testers must perform a number of other tasks common to other IT professionals in order to carry out their roles.

The effectiveness of testing is a major determinant of software quality. It is believed that individual testers vary in their effectiveness, but so far the factors contributing to this variation have not been well studied. In this study, we examined whether personality traits, as described by the five-factor model, affect performance on a software testing task. ICT students were given a small software testing task at which their effectiveness was assessed using several different criteria, including bug location rate, weighted fault density, and bug report quality. Their personality was assessed using the NEO PI-3 personality questionnaire. We then compared testing performance according to individual and aggregate measures against different five-factor personality traits. Several weak correlations between two of these personality traits, extraversion and conscientiousness, and testing effectiveness were found.

Fault based testing is a technique for choosing test cases to reveal certain classes of faults. Due to limited resources and time, testing professionals use their personal experience to (1) "guess" which fault classes are most likely to be present and, then, (2) select appropriate testing methods to reveal such fault classes. The quality of the software depends on whether they can make a good "guess" about the type of faults present and then choose the right testing methods to reveal those faults. However, there is little empirical evidence available in the open literature to support these intuitions. For example, there is no empirical evidence about which types of faults are most commonly made by software developers. By examining the source code changes when faults were fixed in seven open source software artifacts, we propose to classify bug fix patterns into fault classes, and recorded the relative frequencies of those fault classes. This paper reports our findings related to "ifconditional" fixes.

In random testing, the distribution of the generated test cases is known as the test profile. We consider the effects of different test profiles, taking advantage of probabilistic information about likely failure-revealing inputs, on the effectiveness of random testing for debugging. We examine a failure-proportional testing strategy, in which tests are randomly sampled with replacement, with probability proportional to a previously identified failure probability distribution, compared to a uniform testing strategy, in which tests are randomly sampled uniformly from the entire input domain. We show that neither strategy optimises failure-detection capabilities, and show an alternative strategy that does. We also consider selection without replacement, and examine the robustness of some strategies given a divergence between the estimated and actual failure probability distributions.

With the growth of the software testing industry, many in-house testing groups and outsourcing testing companies have been established. Underlying the success of these testing groups and companies are team(s) of testers. This research investigates the importance of different factors, diversity and experience on building a successful testing team. We collected the opinions of testing practitioners on these factors via a survey. The outcome strongly indicates the relative importance of different factors and that diversity is helpful for a testing team. The results also support the importance of suitable team experience.

Metamorphic testing is a technique for the verification of software output without a complete testing oracle. Mathematical optimization, implemented in software, is a problem for which verification can often be challenging. In this paper, we apply metamorphic testing to one such optimization problem, the quadratic assignment problem (QAP). From simple observations of the properties of the QAP, we describe how to derive a number of metamorphic relations useful for verifying the correctness of a QAP solver. We then compare the effectiveness of these metamorphic relations, in "killing" mutant versions of an exact QAP solver, to a simulated oracle. We show that metamorphic testing can be as effective as the simulated oracle for killing mutants. We examine the relative effectiveness of different metamorphic relations, both singly and in combination, and conclude that combining metamorphic relations can be significantly more effective than using a single relation.

Queuing network modelling is a modelling technique for capacity planning studies of computer and communication systems. Due to complexity of the technique, it is very difficult to know from the computed outputs whether the computation of the modelling software is correct. It is necessary to have an effective testing technique to address this problem. Recently, it has been noticed that metamorphic testing is an effective technique for this kind of problem. In this paper, we study the technique of metamorphic testing in testing queuing network modelling through a set of testing experiments in the Java Modelling Tool – a popular open source queuing network modelling suite.

Failure patterns describe typical ways in which inputs revealing program failure are distributed across the input domain—in many cases, clustered together in contiguous regions. Based on these observations several debug testing methods have been developed. We examine the upper bound of debug testing effectiveness improvements possible through making assumptions about the shape, size and orientation of failure patterns. We consider the bounds for testing strategies with respect to minimizing the F-measure, maximizing the P-measure, and maximizing the E-measure. Surprisingly, we find that the empirically measured effectiveness of some existing methods that are not based on these assumptions is close to the theoretical upper bound of these strategies. The assumptions made to obtain the upper bound, and its further implications, are also examined.

Our paper proposes an implementable procedure for using the method of quasi-random sequences in software debug testing. In random testing, the sequence of tests (if considered as points in an -dimensional unit hypercube) will give rise to regions where there are clusters of points, as well as underpopulated regions. Quasi-random sequences, also known as low-discrepancy or low-dispersion sequences, are sequences of points in such a hypercube that are spread more evenly throughout. Based on the observation that program faults tend to lead to contiguous failure regions within a program's input domain, and that an even spread of random tests enhances the failure detection effectiveness for certain failure patterns, we examine the use of quasi-random sequences as a replacement for random sequences in automated testing. Because there are only a small number of quasi-random sequence generation algorithms, and each of them can only generate a small number of distinct sequences, the applicability of quasi-random sequences in testing real programs is severely restricted. To alleviate this problem, we examine the use of two types of randomized quasi-random sequences, which are quasi-random sequences permuted in a nondeterministic fashion in such a way as to retain their low discrepancy properties. We show that testing using randomized quasi-random sequences is often significantly more effective than random testing.

Quasi-random sequences, also known as low-discrepancy or low-dispersion sequences, are sequences of points in an n-dimensional unit hypercube. These sequences have the property that points are spread more evenly throughout the cube than random point sequences, which result in regions where there are clusters of points and others that are sparsely populated. Based on the observation that program faults tend to lead to contiguous failure regions within a program's input domain, and that an even spread of random tests enhances the failure detection effectiveness for certain failure patterns, we examine the use of these sequences as a replacement for random sequences in automated testing.The limited number of quasi-random sequences available from the standard algorithms poses significant practical problems for use when testing real programs, and especially for evaluating its effectiveness. We examine the use of randomised quasi-random sequences, which are permuted in a nondeterministic fashion but still retain their low discrepancy properties, to overcome this problem, and show that testing using randomised quasi-random sequences is often significantly more effective than random testing.