SUMMARY User‐configurable software systems present many challenges to software testers. These systems are created to address a large number of possible uses, each of which is based on a specific configuration. As configurations are made up of groups of configurable elements and settings, a huge number of possible combinations exist. As it is infeasible to test all configurations before release, many latent defects remain in the software once deployed. A just‐in‐time testing technique is presented to address this problem, including examples of how it can be used with various user‐configurable systems in the field. The proposed solution is evaluated with a set of five empirical studies conducted on two separate ABB software systems using real customer configurations and changes. These studies analyse failures reported by many different customers globally and show that this just‐in‐time testing process is effective at detecting latent defects exposed by customer configuration changes in user‐configurable systems. Copyright © 2010 John Wiley & Sons, Ltd.

User-configurable software systems present many challenges to software testers. These systems are created to address a large number of possible uses, each of which is based on a specific configuration. As configurations are made up of groups of configurable elements and settings, a huge number of possible combinations exist. Since it is infeasible to test all configurations before release, many latent defects remain in the software once deployed. An incremental testing process is presented to address this problem, including examples of how it can be used with various user-configurable systems in the field. The proposed solution is evaluated with a set of empirical studies conducted on two separate ABB software systems using real customer configurations and changes. The three case studies analyzed failures reported by many different customers around the world and show that this incremental testing process is effective at detecting latent defects exposed by customer configuration changes in user-configurable systems.

Abstract Testing firewalls have proven to be a useful approach for regression testing in both functional and object‐oriented software. They involve only the modules that are closely related to the changed modules. They lead to substantially reduced regression tests but still are very effective in detecting regression faults. This paper investigates situations when data‐flow paths are longer, and the testing of modules and components only one level away from the changed elements may not detect certain regression faults; an extended firewall considers these longer data paths. We report empirical studies that show the degree to which an extended firewall detected more faults, and how much more testing was required to achieve this increased detection. Copyright © 2008 John Wiley & Sons, Ltd.

Hierons took over from Robin Whitty as Reviews Editor, and from the March 2000 issue forward, all four were simply designated as Editors, although in practice the roles remained the same.We are all very proud of the increased quality of STVR achieved over the years, with substantially increased citations.Derek and I can remember the early days when it was necessary to reject many papers in order to begin to establish that reputation, soliciting papers from our research colleagues in software testing in order to obtain papers to publish.The quality did improve, but the process was frustrating.There are three categories of papers I have had to deal with over the years.• Top notch and well-written papers: these papers were easy to accept and the reviewers were enthusiastic and responsive.• Terrible papers: the reviewers dragged their heels, hated to write reviews and yelled

A testing firewall involves identifying various components that are dependent upon changed elements, but are just one level away from the modified elements. This paper investigates situations when data flow paths are longer, and the mechanism of thorough testing of components one level away from the changed elements may not detect certain regression faults caused by the change; this research has led to the notion of an extended firewall that takes these longer data paths into account. Empirical studies are reported that show the extent to which an extended firewall can detect more faults and how much more testing is required to achieve this increased detection.

Looking for surprises in testing

Industrial real-time systems are complex and need to be thoroughly tested before being released to the customer. We have found that last minute changes are often responsible for the introduction of defects, causing serious problems for the customer. We demonstrate that these defects can be introduced into real-time software in diverse ways, and there is no simple regression testing method that can deal with all of these defect sources. This paper describes the application of a testing firewall for regression testing whose form differs depending upon the defect. The idea of the testing firewall is to limit the regression testing to those potentially affected system elements directly dependent upon changed system elements, and then to thoroughly test these elements. This has resulted in substantial savings in regression testing costs, and yet has been effective in detecting critical defects with significant implication in terms of customer acceptance at ABB. Empirical studies are reported for these experiences in an industrial setting.

Testing graphical user interfaces (GUI) is difficult, involving many states, inputs and events. Another serious problem is that in testing GUI, not all effects created by the testing are observable. We have previously reported a scalable method for testing GUIs based on complete interactions sequences (CIS). A CIS is a sequence of GUI objects and selections that collaborate to produce a response for the user called the responsibility. In this paper we use the CIS foundation to develop a new firewall method for GUI regression testing. This new method is an inherently selective regression approach in which only GUI objects in the firewall need be regression tested. An empirical study of a commercial GUI system is used to illustrate this new method.

Generating test data sets which are sufficiently large to effectively cover all the tests required before a software component can be certified as reliable is a time consuming and error-prone task if carried out manually. A key parameter when testing collections is the size of the collection to be tested: an automatic test generator builds a set of collections containing n elements where n ranges from 0 to n/sub crit/. Data coverage analysis allows us to determine rigorously a collection size such that testing with collections of size > n/sub crit/ does not provide any further useful information, i.e. will not uncover any new faults. We conducted a series of experiments on modules from the C++ Standard Template Library which were seeded with errors. Using a test model appropriate to each module, we generated data sets of sizes up to and exceeding the predicted value of n/sub crit/ and verified that after all collections of size /spl les/n/sub crit/ have been tested, no further errors are discovered. Data coverage was also compared with statement coverage testing and random test data set generation. The three testing techniques were compared for effectiveness at revealing errors compared to the number of test data sets used. Statement coverage testing was confirmed as the cheapest, in the sense that it produces its maximal effect for the smallest number of tests applied, but the least effective technique in terms of numbers of errors uncovered. Data coverage was significantly better than random test generation: it uncovered more faults with fewer tests at every point.

For the testing of container classes and the algorithms or programs that operate on the data in a container, these data have the property of being homogeneous throughout the container. We have developed an approach for this situation called data coverage testing, where automated test generation can systematically generate increasing test data size. Given a program and a test model, it can be theoretically shown that there exists a sufficiently large test data set size N, such that testing with a data set size larger than N does not detect more faults. A number of experiments have been conducted using a set of C++ STL programs, comparing data coverage testing with two other testing strategies: statement coverage and random generation. These experiments validate the theoretical analysis for data coverage, confirming the predicted sufficiently large N for each program.

The importance of empirical work for software engineering papers I have noticed that in recent years there has been a growing insistence that software engineering papers, both in journals and conference proceedings, should include empirical results.I can recall a time when a paper that had new and interesting ideas, submitted to either a journal or a conference, could be deemed worthy of publication on that basis alone.Today, reviewers and editors expect empirical work to be included in papers submitted for publication.This is not only a requisite for research papers, but also for industrial papers as well, although this empirical aspect may well be in the form of case studies.The reason for this is the need for validation of assumptions or other claims, and the verification of results in the paper.In software engineering, as opposed to some other fields in computer science and engineering, it is often not easy to verify your results analytically, and so an empirical study may be the only way to provide such verification.Reviewers do not always insist on a fully-fledged experiment, but do require at least a lightweight empirical study, still carefully thought out and executed.Also, most reviewers will no longer simply accept the results of empirical study or full experiment, but insist upon some sort of statistical analysis to demonstrate statistical significance of the results.I have been somewhat surprised by the necessity for statistical significance, but experimental results that appear compelling often fail to be statistically significant; you can't just eyeball the data and be confident that it couldn't come out that way by coincidence.I tend to be fairly theoretical in my general approach to research, and so years ago I was not trained well in experimental approaches; I was intimidated by the need for experiments and statistical analysis.Now I require that all my students provide some empirical study in their MSc or PhD research, and often insist that full experiments be designed and executed.Recently a friend, Lionel Briand, who is a respected researcher in experimental software engineering, congratulated me on an experimental study that a PhD student of mine had completed, and that I had presented.I was very proud, considering the source of this praise.So I can now claim that any of you can master these experimental and statistical techniques in presenting your ideas, arguing the case for their usefulness and adoption!The three papers in this issue clearly illustrate several approaches to empirical work.Two of the papers provide fully-fledged experiments, carefully designed with thorough analysis: 'Can

Testing graphical user interfaces (GUIs) is difficult, involving many states, inputs and events. We have previously reported a new method for testing GUIs that is scalable and concentrates on user sequences of GUI objects and selections that collaborate, called complete interaction sequences (CIS) and that produce the desired response for the user called the responsibility. In this paper we extend this approach by investigating the use of memory tools to detect missing effects and CIS sequences, investigating interactions between CIS sequences, and providing empirical studies of five different GUI systems.

Editorial: Specification-based testing revisited?The last issue of STVR, 10(4), was a special issue dedicated to specification-based testing and edited by Rob Hierons and John Derrick.Three papers were included in that issue, one by Edwards providing component testing based on flowgraph sequencing information, a second paper by Wimmel et al., indicating how testing can be determined from a state-based specification, and a third by Hong et al., describing how tests can be systematically generated from a statechart, in effect a special form of state-based specification.The interesting observation is that both of the papers in this issue of STVR, 11(1), also involve specification-based testing.For example, the first paper in this issue, by Periyasamy and Alagar, 'A rigorous method for test templates generation from object-oriented specifications', investigates specifications written in Object-Z to generate specification-based tests.Object-Z is an extension of the Z notation that preserves the basic syntax and semantics of Z, but provides additional syntax for features such as inheritance, encapsulation and polymorphism.The primary emphasis of the paper is to generate test templates and oracles for composite operations in an Object-Z specification.A composite operation in Object-Z is specified using the names of operations that are composed using unnamed schemas.These unnamed schemas may introduce additional variables that are subject to scope rules, thus imposing additional constraints on the composed operations.Note that as a result, the input and output spaces are not explicit from the syntax of the composite operation.The correctness of the proposed method is informally justified based upon the semantics of the composite operators in Object-Z.The authors have applied the method to an electronic mail system example.Challenges for future research are to establish scalability and to automate the method with one or more tools.The second paper is written by Bogdanov and Holcombe, 'Statechart testing method for aircraft control systems', and could be considered a companion paper to the one on statecharts by Hong et al. that was included in the special STVR issue 10(4).In the Bogdanov and Holcombe paper, the authors describe a modification of a formal testing method for extended finite state machines (FSMs) to allow the testing of a system based upon statechart specifications.This research was motivated by the fact that a number of concurrent control systems for aircraft have been specified using statecharts.It is interesting that, as a starting point to test simple statecharts, the authors reference and utilize Chow's W method [1].Recently, this Editor used an FSM to partially model a GUI for testing, and Chow's W method was rediscovered for use in testing this FSM model.Readers unfamiliar with the Chow paper might want to check on it for the solution to a commonly occurring problem in testing.Bogdanov and Holcombe have applied their testing method to statechart specifications of part of an autopilot as a case

We explore the use of multivariate visualization techniques to support a new approach to test data selection, called observation-based testing. Applications of multivariate visualization are described, including: evaluating and improving synthetic tests; filtering regression test suites; filtering captured operational executions; comparing test suites; and assessing bug reports. These applications are illustrated by the use of correspondence analysis to analyze test inputs for the GNU GCC compiler.

All aspects of software engineering strive toward guaranteeing that an implemented algorithm performs the intended tasks. The paper describes results in investigating the relationship between specifications and re-engineered code for data processing programs in COBOL, and develops strategies for assigning significance to identified differences. A modified F(p) notation representation of the program will be manipulated into a standard form, from which features called program points will be used to guide complexity reduction. Finally, backbone elements called threads will be identified to explore the equivalence between the specifications and the program, where each thread will represent a different behavior mode of the program.

Testing graphical user interfaces (GUI) is a difficult problem due to the fact that the GUI possesses a large number of states to be tested, the input space is extremely large due to different permutations of inputs and events which affect the GUI, and complex GUI dependencies which may exist. There has been little systematic study of this problem yielding a resulting strategy which is effective and scalable. The proposed method concentrates upon user sequences of GUI objects and selections which collaborate, called complete interaction sequences (CIS), that produce a desired response for the user. A systematic method to test these CIS utilizes a finite-state model to generate tests. The required tests can be substantially reduced by identifying components of the CIS that can be tested separately. Since consideration is given to defects totally within each CIS, and the components reduce required testing further, this approach is scalable. An empirical investigation of this method shows that substantial reduction in tests can still detect the defects in the GUI. Future research will prioritize testing related to the CIS testing for maximum benefit if testing time is limited.

The maturity of object-oriented methods has led to the wide availability of container classes: classes that encapsulate classical data structures and algorithms. Container classes are included in the C++ and Java standard libraries, and in many proprietary libraries. The wide availability and use of these classes makes reliability important, and testing plays a central role in achieving that reliability. The large number of cases necessary for thorough testing of container classes makes automated testing essential. This paper presents a novel approach for automated testing of container classes based on combinatorial algorithms for state generation. The approach is illustrated with black-box and white-box test drivers for a class implemented with the red–black tree data structure, used widely in industry and, in particular, in the C++ Standard Template Library. The white-box driver is based on a new algorithm for red–black tree generation. The drivers are evaluated experimentally, providing quantitative measures of their effectiveness in terms of block and path coverage. The results clearly show that the approach is affordable in terms of development cost and execution time, and effective with respect to coverage achieved. The results also provide insight into the relative advantages of black-box and white-box drivers, and into the difficult problem of infeasible paths. Copyright © 2000 John Wiley & Sons, Ltd.

Structural coverage approaches to software testing are mature, having been thoroughly studied for decades. Significant tool support, in the form of instrumentation for statement or branch coverage, is available in commercial compilers. While structural coverage is sensitive to which code structures are covered, it is insensitive to the values of the variables when those structures are executed. Data coverage approaches, e.g. boundary value coverage, are far less mature. They are known to practitioners mostly as a few useful heuristics with very little support for automation. Because of its sensitivity to variable values, data coverage has significant potential, especially when used in combination with structural coverage. This paper generalizes the traditional notion of boundary coverage, and formalizes it with two new data coverage measures. These measures are used to generate test cases automatically and from these, sophisticated test suites for functions from the C++ Standard Template Library. Finally, the test suites are evaluated with respect to both structural coverage and discovery of seeded faults. Copyright © 1999 John Wiley & Sons, Ltd.

Graphics user interface (GUI) design is an expected part of almost every contemporary software or computer project, and yet the testing is usually approached in an ad hoc fashion. The reason GUI testing is different and difficult is that the input is interactive whereas the output may be graphical or may be an event. An especially serious problem occurs in maintenance where changes are made to the GUI interface, and the maintenance engineer does not have a sufficient understanding of how the GUI software was designed; in this case, an automated GUI testing method is needed. The paper identifies both static and dynamic event interactions in a GUI, and shows how automated regression tests can be generated to test these interactions. It is shown how Latin Squares can be used for this automated test design, where Latin Squares techniques are borrowed from statistical experimental design. Three methods are compared: brute force test generation, random test generation and the use of Latin Squares, comparing the number of tests required to cover specified GUI event interactions in each case.

H. Leung and L. White (1990) introduced the firewall concept for control flow in regression testing. In order to define this firewall, they assumed that unit testing and integration testing are both reliable. The paper continues to assume that unit testing can be carried out in an entirely reliable fashion, while integration testing may not be reliable. The paper is designed to fulfill the following objectives: identify the error cases that the testing process may have missed, making integration testing unreliable; attempt to solve the problems that make integration testing unreliable, designing procedures to improve reliability; and determine the effects of unreliable integration testing on the firewall, indicating how the firewall construction should subsequently change.

Too Many Conference Choices?In the past five years, many new conferences of interest to researchers and practitioners in the area of software testing, verification and reliability have been introduced.As will be shown, this represents a significant dilemma in terms of which conferences to attend, and also there is the additional expense of attending multiple conferences throughout the year.First, let us consider the general software engineering conferences of interest to our community.In order to construct the following lists of conferences, I have borrowed from the excellent S W R Diary compiled in each issue for us by Martin Woodward.However, I accept total responsibility for any omissions of important conferences or workshops; such omissions are almost inevitable.Another point to note is that in some cases conferences already held have been included, rather than indicating the next in the series.

CWRU has participated in a joint project with IBM in developing an internal regression testing tool called Test Manager. The objective of Test Manager is to aid in regression testing at the unit testing, integration testing, or function testing levels and also to produce a reduced regression test set to verify the software changes indicated by the user. A conceptual basis for the tool, called a firewall, and a description of the factors which influenced the design of the tool are given. The tool utilizes a Prolog database; a set of experiments that determined how that database should best be structured is described. Timing and space requirements and future work and enhancements to the tool are discussed.>

This paper provides an approach to test nonlinear functions in computer programs, whether this function is used for control flow, such as a predicate inequality or equality constraint, or is given as an input-output relationship. This approach will obtain test data to detect linear errors in the given nonlinear function. An error-space criterion previously given by Zeil will be utilized, and a necessary and sufficient condition for the test data will be specified to guarantee the satisfaction of this criterion. This leads to a simple and efficient method to select test data which satisfies that condition; only (n+2) tests are required, where n is the number of input variables. An analysis will be given to show that this simple approach can be very effective in detecting nonlinear errors as well.

The authors present a methodology for regression testing and function or system testers. The methodology involves regression testing of modules where dependencies due to both control flow and data flow are taken into account. The control-flow dependency is modeled as a call graph and a firewall defined to include all affected modules which must be retested. Global variables are considered as the remaining data-flow dependency to be modeled. An approach to testing and regression testing of these global variables is given, and a firewall concept for the data-flow aspect of software change is defined.>

Domain testing attempts to find errors in the numeric expressions affecting the flow of control through a program. Intuitively, domain testing provides a systematic form of boundary value testing for the conditional statements within a program. Several forms of domain testing have been proposed, all dealing with the detection of linear errors in linear functions. Perturbation analysis has been previously developed as a measure of the volume of faults, from within a selected space of possible faults, left undetected by a test set. It is adapted here to errors and error spaces. The adapted form is used to show that the different forms of domain testing are closer in error detection ability than had been supposed. They may all be considered effective for finding linear errors in linear predicate functions. A simple extension is proposed, which allows them to detect linear errors in nonlinear predicate functions using only a single additional test point.

A software test cost model which can be used to compare the retest-all strategy to a selective strategy is proposed. A test cost model is presented, and the conditions under which the selective strategy is more economical than the retest-all strategy are identified. Recently introduced regression testing strategies are reviewed. It is found that the relative cost depends on four factors: the selection, the execution, the checking cost, and the ratio of the size of the selected previous test to the size of the total previous test.>

Abstract The purpose of this paper is to present a method for testing computer programs with iteration loops. Given such programs, we have shown that for classes of program paths, identified assequences of simple loop paths, there is a characterizing function called asimple loop pattern. The key idea of simple loop patterns is that these special functions form a base set which can represent any path computation in the given program. A software tool called SILOP has been developed to automatically generate these simple loop patterns, and each corresponding sequence of simple loop paths can be considered as a test case. The tester uses each test case, and with knowledge of the application program, can generate corresponding test data. This paper also presents a method for selecting the specific paths and test data to determine the simple loop pattern reliably. The tester can use this selection method to predict the number of tests required. In order to apply this selection method, the given program must be alinear computer program. The SILOP tool and this test selection method have been applied to commercial software; in this paper, this computational experience is reported and several examples are given to demonstrate the approach.

The authors identify the common errors and faults in combining modules into a working unit. They also make practical recommendations on test selection for integration testing and utilize these recommendations for regression testing at the integration level. With emphasis on reusing the previous test cases and retesting only the parts that are modified, one can reduce the testing expenses. The concept of 'firewall' is proposed to assist the tester in focusing on that part of the system where new errors may have been introduced by a correction or a design change. An experiment is presented in which an application of strategy is compared with the 'retest-all' strategy. By applying the approach for test selection, it is possible to discover all errors found by the retest-all strategy by executing only 35% of the total number of test cases.>

Abstract Despite the existence of global variables in many software systems, testing global variables has traditionally been overlooked by the testing community. This state of the practice leads to the release of some software systems with untested properties. In this paper, we present some insights into the problem of testing and regression testing global variables. A basis set of testing problems for parameters will be identified and the testing problem of global variables will be mapped onto a combination of these basis cases. It will be shown that global variables can be treated as parameters and can be tested accordingly. Testing strategies and regression testing strategies for unit, integration, and system testing global variables are described.

A problem facing those conducting maintenance testing is to identify proper test classes. The notion of regression testability is introduced as a way to measure the ease of retesting. Both the program design and the test plan design may affect the regression testability of a program. The testing set, testing number, regression number and stable and workable metrics are developed to measure the program and the test plan and program testability is defined. The retest strategy is proposed for performing corrective regression testing. This approach involves the use of the test plan in a way that will reduce the amount of testing required. The guiding principle of the retest strategy is to view the regression testing problem as being composed of two subproblems (the test selection problem and the test plan update problem). To this end, the author's retest strategy consists of two major phases (the test classification phase and test plan update phase).>

This paper presents a testing strategy desiged to detect errors in the control flow of a computer program, and the conditions under which this strategy is reliable are given and characterized. The control flow statements in a computer progam partition the input space into a set of mutually exclusive domains, each of which corresponds to a particular program path and consists of input data points which cause that path to be executed. The testing strategy generates test points to examine the boundaries of a domain to detect whether a domain error has occurred, as either one or more of these boundaries will have shifted or else the corresponding predicate relational operator has changed. If test points can be chosen within e of each boundary, under the appropriate assumptions, the strategy is shown to be reliable in detecting domain errons of magnitude greater than ∈. Moreover, the number of test points required to test each domain grows only linearly with both the dimensionality of the input space and the number of predicates along the path being tested.