Constraints can be found in many specifications of a software system. The impact of constraints varies with the test problem, but their presence causes problems for many existing combinatorial interaction testing (CIT) tools. Of the numerous existing tools supporting CIT design only a few offer full constraints support. Of these few tools those with full published details are even rarer. In extension to existing Boolean constraints we propose numerical constraints. We discuss definition, usage and handling in this work and integrate results with the classification tree method.

The combinatorial test design and combinatorial interaction testing are well studied topics. For the generation of dynamic test sequences from a formal specification of combinatorial problems, there has not been much work yet. The classification tree method implements aspects from the field of combinatorial testing. This paper extends the classification tree with additional information to allow the interpretation of the classification tree as a hierarchical concurrent state machine. Using this state machine, our new approach then uses a Multi-agent System to generate test sequences by finding and rating valid paths through the state machine.

Evolutionary structural testing has been researched and promising results have been presented. However, it has hardly been applied to real-world complex systems and as such, little is known about the scalability, applicability and acceptability of it in an industrial setting. The European project EvoTest (IST-33472) team has been working from 2006 till 2009 to improve this situation and this paper informs about the results. We start with an overview of tools and techniques which we have developed for automated evolutionary structural testing. Subsequently, we describe the empirical setup used to study the applicability of evolutionary structural testing in industry through two case studies. The test objects used for the studies are selected functions (handwritten and generated) from production systems at Daimler and Berner & Mattner Systemtechnik (BMS) like, for example, Rear Window Defroster, Global Powertrain Engine Controller, Window Lift Control System, etc. The results of the case studies are described and research questions are assessed based on the obtained results. In summary, the results indicate that evolutionary structural testing in an industrial setting is worthwhile and profitable. Hardly any detailed knowledge of evolutionary computation is required to search for interesting test data. The case studies also research the benefits of using techniques like automated parameter tuning and search space smoothing.

Search-Based Test Data Generation reformulates testing goals as fitness functions so that test input generation can be automated by some chosen search-based optimization algorithm. The optimization algorithm searches the space of potential inputs, seeking those that are “fit for purpose,” guided by the fitness function. The search space of potential inputs can be very large, even for very small systems under test. Its size is, of course, a key determining factor affecting the performance of any search-based approach. However, despite the large volume of work on Search-Based Software Testing, the literature contains little that concerns the performance impact of search space reduction. This paper proposes a static dependence analysis derived from program slicing that can be used to support search space reduction. The paper presents both a theoretical and empirical analysis of the application of this approach to open source and industrial production code. The results provide evidence to support the claim that input domain reduction has a significant effect on the performance of local, global, and hybrid search, while a purely random search is unaffected.

Evolutionary testing has been researched and promising results have been presented. However, evolutionary testing has remained predominately a research-based activity not practiced within industry. Although attempts have been made, such as Daimler's Evolutionary Structural Test (EST) prototype, until now, no such tool has been suitable for industrial adoption. The European project EvoTest (IST-33472) team has been working from 2006 till 2009 to improve this situation. This paper describes the final version of the Evolutionary Testing Framework (ETF) resulting from the EvoTest project. In specific we will present the EvoTest Structural Testing tool for fully automatic structural testing that has been demonstrated to be suitable within an industrial setting. The paper concentrates on how to use it and interpret the results. The paper starts with introducing the concepts of Evolutionary Testing in general and Structural Testing in specific. Subsequently, the ETF and the EvoTest Structural Testing tool built on-top of it will be described. We will concentrate on the usage, the architecture, and remaining limitations of the tool. The paper concludes describing the results of using the EvoTest Structural Testing tool in practice on real-world systems in an industrial setting.

During the development of electronic control units (ECU) in domains like the automotive industry, tests are performed on various test platforms, such as model-in-the-loop, software-in-the-loop, processor-in-the-loop, and hardware-in-the-loop platforms in order to find faults in early development stages. Test cases must be specified to verify the properties demanded of the system on these test platforms. This is an expensive and non-trivial task. Evolutionary black-box testing, a recent approach to automating the creation of interesting test cases, can solve this task completely automatically.

Evolutionary structural testing is an approach to automatically generating test cases that achieve high structural code coverage. It typically uses genetic algorithms (GAs) to search for relevant test cases. In recent investigations particle swarm optimization (PSO), an alternative search technique, often outperformed GAs when applied to various problems. This raises the question of how PSO competes with GAs in the context of evolutionary structural testing. In order to contribute to an answer to this question, we performed experiments with 25 small artificial test objects and 13 more complex industrial test objects taken from various development projects. The results show that PSO outperforms GAs for most code elements to be covered in terms of effectiveness and efficiency.

There has recently been a great deal of interest in search-based test data generation, with many local and global search algorithms being proposed. However, to date, there has been no investigation ofthe relationship between the size of the input domain (the search space) and performance of search-based algorithms. Static analysis can be used to remove irrelevant variables for a given test data generation problem, thereby reducing the search space size. This paper studies the effect of this domain reduction, presenting results from the application of local and global search algorithms to real world examples. This provides evidence to support the claimthat domain reduction has implications for practical search-based test data generation.

Timing constraints in embedded systems must be satisfied so that real-time embedded systems work properly and safely. Execution time testing involves finding the best and worst case execution times. Evolutionary testing is used to dynamically search for the extreme execution times. During the evolutionary search, some parts of the source code are never accessed. Moreover, it turns out that the search delivers different extreme execution times in a high number of generations. We propose a new approach which makes use of seeding the evolutionary algorithm with test data achieved a high structural coverage. This new method leads to raise the confidence in the results and to gain in efficiency in terms of number of generations needed.

Evolutionary algorithms have been successfully applied in the area of software testing. However, previous approaches in the area of object-oriented testing are limited in terms of test case feasibility due to call dependences and runtime exceptions. In this paper, we present a search-based approach to automatically generating test cases for object-oriented software. It relies on a tree-based representation of method call sequences. Strongly-typed genetic programming is employed to generate method call trees which respect the call dependences among the methods. We apply a new kind of distance-based fitness function that accounts for runtime exceptions. In a case study, the approach outperformed random testing in terms of achieved coverage and it produced test cases achieving full branch coverage for a test object that makes ample use of explicit runtime exceptions.

Evolutionary algorithms have successfully been applied to software testing. Not only approaches that search for numeric test data for procedural test objects have been investigated, but also techniques for automatically generating test programs that represent object-oriented unit test cases. Compared to numeric test data, test programs optimized for object-oriented unit testing are more complex. Method call sequences that realize interesting test scenarios must be evolved. An arbitrary method call sequence is not necessarily feasible due to call dependences which exist among the methods that potentially appear in a method call sequence. The approach presented in this paper relies on a tree-based representation of method call sequences by which sequence feasibility is preserved throughout the entire search process. In contrast to other approaches in this area, neither repair of individuals nor penalty mechanisms are required. Strongly-typed genetic programming is employed to generate method call trees. In order to deal with runtime exceptions, we use an extended distance-based fitness function. We performed experiments with four test objects. The initial results are promising: high code coverages were achieved completely automatically for all of the test objects.

Like other engineering disciplines, software engineering is typically concerned with near optimal solutions or those which fall within a specified applicable tolerance. More recently, search-based techniques have started to find application in software engineering problem domains. This area of search-based software engineering has its origins in work on search-based testing, which began in the mid 1990s. Already, search-based solutions have been applied to software engineering problems right through the development life cycle.

A testability transformation is a source-to-source transformation that aims to improve the ability of a given test generation method to generate test data for the original program. We introduce testability transformation, demonstrating that it differs from traditional transformation, both theoretically and practically, while still allowing many traditional transformation rules to be applied. We illustrate the theory of testability transformation with an example application to evolutionary testing. An algorithm for flag removal is defined and results are presented from an empirical study which show how the algorithm improves both the performance of evolutionary test data generation and the adequacy level of the test data so-generated.

Evolutionary testing is based on the principle of searching for relevant test cases in the input domain of the system under test with the help of evolutionary algorithms. Evolutionary testing enables the complete automation of test case design whenever the test aim can be expressed numerically, e.g. when performing temporal behavior testing, safety testing, or structural testing. Evolutionary tests have already produced very good results in all of these application fields. Due to the full automation of evolutionary testing, the effectiveness and efficiency of the test can clearly be improved. The system could be tested with a large number of different input situations. In most cases, more than several thousand test data sets are generated and executed within a few minutes.

Evolutionary testing is a search based approach to the automated generation of systematic test data, in which the search is guided by the test data adequacy criterion. Two problems for evolutionary testing are the large size of the search space and structural impediments in the implementation of the program which inhibit the formulation of a suitable fitness function to guide the search. In this paper we claim that slicing can be used to narrow the search space and transformation can be applied to the problem of structural impediments. The paper presents examples of how these two techniques have been successfully employed to make evolutionary testing both more efficient and more effective.

Side-effects are widely believed to impede program comprehension and have a detrimental effect upon software maintenance. This paper introduces an algorithm for side-effect removal which splits the side-effects into their pure expression meaning and their state-changing meaning. Symbolic execution is used to determine the expression meaning, while transformation is used to place the state-changing part in a suitable location in a transformed version of the program. This creates a program which is semantically equivalent to the original but guaranteed to be free from side-effects. The paper also reports the results of an empirical study which demonstrates that the application of the algorithm causes a significant improvement in program comprehension.

For real-time systems, correct system functionality depends on logical as well as on temporal correctness. Static analysis alone is not sufficient to verify the temporal behavior of real-time systems. Since existing test methods are not specialized for the verification of temporal correctness, we have developed a new testing method, namely evolutionary testing. This paper illustrates results of the first industrial application of the evolutionary test.