A number of authors hypothesize and experimentally confirm that Cyclomatic Complexity (CC) has a very strong correlation with Lines of Code (LOC), justifying the use of LOC in place of CC. Others report on a moderate correlation and advocate for the use of both metrics. These studies have, for the most part, studied production code, and we suspect different results may be observed for test code. With 40 different, large open-source subjects and five subjects from industry partners, we collected metric values for LOC, CC and Halstead Effort (HE) and measured their correlation. In test code, contrary to production code, there exist a very weak (or almost no) correlation between (a) LOC and CC, and weak (nearly moderate) correlation for (b) HE and CC, and (c) LOC and HE. We therefore argue and propose that the level of correlation depends on at least three factors namely: the kind of code (i.e., production code vs test code), the kind of software (open-source vs industry) and the kind of metric (LOC, CC, HE). Given the weak monotonicity between CC, LOC and HE we observe, we aspire to challenge the viewpoint that CC and Halstead metrics are redundant with LOC, as some studies suggest, at least on test code. We therefore advocate for using CC over LOC (or both, or cyclomatic density) when studying test code, as CC is perceived to better reflect cognitive complexity, numerical complexity, interdependency and code refactoring that cannot be accounted for simply by LOC.

Background. Testing is an essential activity in the software development life cycle. Nowadays, testing activities are widely spread along the software development process, since software products are continuously tested to meet the user's expectations and to compete in global markets. In this context, internationalization testing is defined as the practice focused on determining that a software works properly in a specific language and in a particular region. Aims. This study aims to explore the particularities of internationalization testing in the software industry and discuss the importance of this practice from the point of view of professionals working in this context. Method. We developed an exploratory qualitative study and conducted interviews with professionals from an international software company, in order to understand three aspects of internationalization testing: general characteristics and importance of this practice, particularities of the process, and the role of test automation in this context. Results. An amount of 13 professionals participated in this study. Results demonstrated that internationalization testing is mostly related to aspects of graphical user interfaces. In this context, truncation and mistranslations are the main faults observed, and test automation might be difficult to implement and maintain due to amount validations that are human-dependent. Conclusion. Internationalization testing is an important practice to guarantee the quality of software products developed for global markets. However, this aspect of software testing remains unpopular or unfamiliar among professionals. This study is a step forward in the process of informing and enlightening academic researchers and practitioners in industry about this theme.

The issue of validity threats in empirical software engineering research is important. However, some authors overlook this, focusing on validating their work through application of fundamental testing techniques, instead. However, testing is different to empirical validation, with the latter being more concerned about how experimental conclusions are justified. An important factor that can render an experimental conclusion incorrect is researcher's bias, which can be especially relevant when setting the experimental parameters. Therefore, consideration of validity threats is essential to enable confidence in research results and assure the research quality. This paper provides a practical approach for mitigating threats to validity in empirical software engineering using a sequence of software activities. The paper is based on a real-world traceability case study for illustration purposes.

In Systems Engineering (SE), development of complex systems involves a collaboration of expertise from different domains. Heterogeneous artifacts are generated using different modeling tools. Capturing the traceability information among these artifacts helps serve many purposes, including change impact analysis; validation and verification; and requirements tracking. However, creating trace links among these heterogeneous artifacts is problematic. No precise semantics exist for the trace links that relate them. This paper shows how to capture traceability information in a system with heterogeneous artifacts, illustrated here using an avionics case study that uses a traceability model and a trace links taxonomy that we constructed and published previously.

Context: UML software development relies on different types of UML diagrams, which must be consistent with one another. UML Synthesis techniques suggest to generate diagram(s) from other diagram(s), thereby implicitly suggesting that input and output diagrams of the synthesis process be consistent with one another.

Many companies, including Ericsson, experience increased software verification costs. Agile cross-functional teams find it easy to make new additions of test cases for every change and fix. The consequence of this phenomenon is duplications of test code. In this paper, we perform an industrial case study that aims at better understanding such duplicated test fragments or as we call them, clones. In our study, 49% (LOC) of the entire test code are clones. The reported results include figures about clone frequencies, types, similarity, fragments, and size distributions, and the number of line differences in cloned test cases. It is challenging to keep clones consistent and remove unnecessary clones during the entire testing process of large-scale commercial software.

Abstract To ensure software is performing as intended, it can be black‐box or white‐box tested. Category partition is a black‐box, specification‐based testing technique that begins by identifying the parameters, categories (characteristics of parameters), and choices (acceptable values for categories). These choices are then combined to form test frames on the basis of various criteria such as Base Choice and Each Choice. To ensure that the combinations of choices are feasible, constraints on choices are introduced. Combining choices, while accounting for constraints, to form an each choice adequate test set is feasible (eg, using constrained covering arrays from combinatorial testing). However, the Base Choice criterion has not been defined to specifically account for constraints on choices, resulting in adverse consequences. In this paper, we introduce two extensions to the Base Choice criterion, namely, Constrained Base Choice and Extended Constrained Base Choice to specifically account for (complex) constraints on choices. We use a number of academic and industrial case studies to compare different adequacy criteria, including the new ones, in terms of cost and effectiveness at finding faults. Results show the performance of the new criteria equivalent to a 3‐way combination criterion with a much smaller cost.

The practitioner interested in reducing software verification effort may found herself lost in the many alternative definitions of Graphical User Interface (GUI) testing that exist and their relation to the notion of system testing . One result of these many definitions is that one may end up testing the same parts of the Software Under Test (SUT), specifically the application logic, twice. To clarify two important testing activities and avoid duplicate testing effort, this paper empirically evaluates to what extent GUI tests exercise the application logic of the software under test (and not only the GUI code). Experimental results show that GUI tests do not necessarily entirely exercise application logic functionality, at least not as much as system tests directly interacting with application logic code.

State diagrams are widely used to model software artifacts, making state-based testing an interesting research topic. When conducting research on state-based testing for evaluating different testing criteria, often there is a need to devise numerous test suites in a systematic way according to selection criteria such as all-edges, all-transition-pairs, or the transition tree (W-method). Moreover, one also needs to satisfy each criterion in as many ways as possible to account for possible stochastic phenomena within each criterion. The main issue is then: how to automate the generation of as many, or even all, the different test suites for each criterion? This paper presents the first part of a framework, an automation tool chain that generates test trees from a state machine diagram, extracts test cases from the generated trees, and composes a test suite from each generated tree. This tool is the first to generate all possible distinctive trees using depth and breadth first graph traversal algorithms. The tool chain should be of interest to researchers in state-based testing as well as practitioners who are interested in alternative adequate test suites especially for comparing the effectiveness of the different test suites satisfying one criterion and the effectiveness of the other different criteria.

This paper proposes an algorithm that is particularly concerned with generating all possible distinct spanning trees that are based on breadth-first-search directed graph traversal. The generated trees span all edges and vertices of the original directed graph. The algorithm starts by generating an initial tree, and then generates the rest of the trees using elementary transformations. It runs in O(E+T) time where E is the number of edges and T is the number of generated trees. In the worst-case scenario, this is equivalent to O (E+En/Nn) time complexity where N is the number of nodes in the original graph. The algorithm requires O(T) space. However, possible modifications to improve the algorithm space complexity are suggested. Furthermore, experiments are conducted to evaluate the algorithm performance and the results are listed.

Traceability provides a mean for Software Engineers to track system artifacts at different levels of abstraction to verify and validate system requirements. This paper provides a systematic literature review about modeling traceability in computer systems, particularly, systems that involve artifacts that come from different domains of expertise (i.e., heterogeneous artifacts). Our findings show that there is a lack of research that focus on modeling traceability among heterogeneous artifacts, which reflects in inadequate traceability tools, and that precise semantics for trace links among artifacts is needed. Our findings lead us to highlight the key areas that can enhance research on those directions.

Finite state machines being intuitively understandable and suitable for modeling in many domains, they are adopted by many software designers. Therefore, testing systems that are modeled with state machines has received genuine attention. Among the studied testing strategies are complete round-trip paths and transition trees that cover round-trip paths in a piece wise manner. We present an empirical study that aims at comparing the effectiveness of the complete round-trip paths test suites to the transition trees test suites in one hand, and comparing the effectiveness of the different techniques used to generate transition trees (breadth first traversal, depth first traversal, and random traversal) on the other hand. We also compare the effectiveness of all the testing trees generated using each single traversal criterion. This is done through conducting an empirical evaluation using four case studies from different domains. Effectiveness is evaluated with mutants. Experimental results are presented and analyzed.

Category Partition (CP) is a black box testing technique that formalizes the specification of the input domain in a CP specification for the system under test. A CP specification is driven by tester's expertise and bundles parameters, categories (characteristics of parameters) and choices (acceptable values for categories) required for extensively testing the system. For completeness the choices correspond to permitted input values as well as some values to account for boundaries or robustness. These choices are then combined to form test frames on the basis of various criteria such as each choice or pairwise. To ensure that the combinations of choices are feasible and account for valid sets of user requirements, constraints are introduced to specify permitted combinations among choices, and to specify single or error choices. In a typical development environment where testing is driven by stringent deadlines a tester might have to decide how many constraints are enough to attain the maximum level of test completeness. The present work will assist a test engineer in making this decision. We conclude, on the basis of our experimental evaluation on academic and industrial case studies, that an equally effective test suite can be attained by meticulously defining error and single annotations in a CP specification while ignoring other constraints among choices.

Avionics systems, along with their internal hardware and software components interfaces, must be well defined and specified (e.g., unambiguous, complete, verifiable, consistent, and traceable specification). Such a specification is usually written in the form of an Interface Control Document (ICD), and represents the cornerstone of the avionics system integration activities. However, there is no commonly accepted language to define and use these ICDs and no common definition of what an ICD is or should contain. Indeed, avionics companies define their own, proprietary ICDs and processes. In this paper, we first identify the pieces of information that an ICD should contain for both federated and IMA open systems. Then, we propose a data extraction process that enables better understanding and more efficient extraction of open avionics systems interface specifications, and provides a clearer vision on the information needed to build a model driven solution for modeling avionics system interfaces, our long-term goal. We validate this process by applying it on a set of open avionics sub-system standards and the results have shown its feasibility.

Verifying a real time embedded application is challenging since one has to consider timing requirements in addition to functional ones. During online state-based testing the generation and execution of test cases happen concurrently: test case generation uses information from a state-based test model in combination with observed execution behaviour. This paper describes a practical online testing algorithm that is implemented in the state-based modeling tool RTEdge. Two case studies show that our online testing algorithm produces a test suite that achieves high model coverage, thus facilitating the automated verification of real-time embedded software.

Traceability links, which relate artifacts created during system development, are important for assuring product quality. Although literature provides many techniques to model traceability, existing solutions are either tailored to specific domains (e.g., Ecore modeling languages), or not complete enough (e.g., lack support to specify traceability link semantics). We propose a model that is not domain specific; it provides a solution for modeling traceability links among heterogeneous models, that is, systems for which traceability links need to be established between artifacts in widely different modeling languages (e.g., UML, block diagrams, informal documents). Our solution incorporates, in one model, ideas from many existing solutions, in an attempt to be as complete as possible. We also argue that our solution is extensible in the sense that it can adapt to new modeling languages, new ways of characterizing traceability information for instance, without requiring changes to the model itself.

Summary This paper presents a case study of coverage‐based regression testing techniques on a real world industrial system with real regression faults. The study evaluates four common prioritization techniques, a test selection technique, a test suite minimization technique and a hybrid approach that combines selection and minimization. The study also examines the effects of using various coverage criteria on the effectiveness of the studied approaches. The results show that prioritization techniques that are based on additional coverage with finer grained coverage criteria perform significantly better in fault detection rates. The study also reveals that using modification information in prioritization techniques does not significantly enhance fault detection rates. The results show that test selection does not provide significant savings in execution cost (<2%), which might be attributed to the nature of the changes made to the system. Test suite minimization using finer grained coverage criteria could provide significant savings in execution cost (79.5%) while maintaining a fault detection capability level above 70%, thus representing a possible trade‐off. The hybrid technique did not provide a significant improvement over traditional minimization techniques. Copyright © 2015 John Wiley & Sons, Ltd.

The transition from an informal requirements specification in natural language to a structured, precise specification is an important challenge in practice. It is particularly so for object-oriented methods, defined in the context of the OMG's Model Driven Architecture (MDA), where a key step is to transition from a use case model to an analysis model. However, providing automated support for this transition is challenging, mostly because, in practice, requirements are expressed in natural language and are much less structured than other kinds of development artifacts. Such an automated transformation would enable at least the generation of an initial, likely incomplete, analysis model and enable automated traceability from requirements to code, through various intermediate models. In this article, we propose a method and a tool called aToucan, building on existing work, to automatically generate a UML analysis model comprising class, sequence and activity diagrams from a use case model and to automatically establish traceability links between model elements of the use case model and the generated analysis model. Note that our goal is to save effort through automated support, not to replace human abstraction and decision making. Seven (six) case studies were performed to compare class (sequence) diagrams generated by aToucan to the ones created by experts, Masters students, and trained, fourth-year undergraduate students. Results show that aToucan performs well regarding consistency (e.g., 88% class diagram consistency) and completeness (e.g., 80% class completeness) when comparing generated class diagrams with reference class diagrams created by experts and Masters students. Similarly, sequence diagrams automatically generated by aToucan are highly consistent with the ones devised by experts and are also rather complete, for instance, 91% and 97% message consistency and completeness, respectively. Further, statistical tests show that aToucan significantly outperforms fourth-year engineering students in this respect, thus demonstrating the value of automation. We also conducted two industrial case studies demonstrating the applicability of aToucan in two different industrial domains. Results showed that the vast majority of model elements generated by aToucan are correct and that therefore, in practice, such models would be good initial models to refine and augment so as to converge towards to correct and complete analysis models. A performance analysis shows that the execution time of aToucan (when generating class and sequence diagrams) is dependent on the number of simple sentences contained in the use case model and remains within a range of a few minutes. Five different software system descriptions (18 use cases altogether) were performed to evaluate the generation of activity diagrams. Results show that aToucan can generate 100% complete and correct control flow information of activity diagrams and on average 85% data flAow information completeness. Moreover, we show that aToucan outperforms three commercial tools in terms of activity diagram generation.

Context: The Unified Modeling Language (UML), with its 14 different diagram types, is the de-facto standard modeling language for object-oriented modeling and documentation. Since the various UML diagrams describe different aspects of one, and only one, software under development, they are not independent but strongly depend on each other in many ways. In other words, the UML diagrams describing a software product must be consistent. Inconsistencies between these diagrams may be a source of faults in software systems. It is therefore paramount that these inconsistencies be detected, analyzed and hopefully fixed.

In this paper we propose a method and a tool to generate test suites from extended finite state machines, accounting for multiple (potentially conflicting) objectives. We aim at maximizing coverage and feasibility of a test suite while minimizing similarity between its test cases and minimizing overall cost. Therefore, we define a multi-objective genetic algorithm that searches for optimal test suites based on four objective functions. In doing so, we create an entire test suite at once as opposed to test cases one at a time. Our approach is evaluated on two different case studies, showing interesting initial results.

For functional testing based on the input domain of a functionality, parameters and their values are identified and a test suite is generated using a criterion exercising combinations of those parameters and values. Since software systems are large, resulting in large numbers of parameters and values, a technique based on combinatorics called Combinatorial Testing (CT) is used to automate the process of creating those combinations. CT is typically performed with the help of combinatorial objects called Covering Arrays. The goal of the present work is to determine available algorithms/tools for generating a combinatorial test suite. We tried to be as complete as possible by using a precise protocol for selecting papers describing those algorithms/tools. The 75 algorithms/tools we identified are then categorized on the basis of different comparison criteria, including: the test suite generation technique, the support for selection (combination)criteria, mixed covering array, the strength of coverage, and the support for constraints between parameters. Results can be of interest to researchers or software companies who are looking for a CT algorithm/tool suitable for their needs.

The 5th IEEE International Conference on Software Testing, Verification, and Validation (ICST 2012) was held on 17–21 April 2012, in Montreal, Québec, Canada. ICST has established itself as the premier forum for presentation of leading edge results on outstanding topics in all areas related to software quality. The conference brings together researchers and practitioners who study the theory, techniques, technologies, and applications of software testing, verification, and validation. ICST 2012 originally attracted 177 submissions, among which 41 research papers and seven industry papers were selected for inclusion in the proceedings. All papers were refereed by at least three members of the ICST 2012 Program Committee. Of the 48 papers accepted, five papers were selected by the Program Co-Chairs, Antonia Bertolino and Yvan Labiche, for consideration for this special issue of STVR. These papers were extended from their conference version by the authors and subjected to the rigorous STVR reviewing process, thus undergoing additional rounds of reviews and revisions. Three papers successfully completed the review process and are contained in this special issue. The first paper, “Automatic Testing of GUI-Based Applications” by Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro, describes a technique to automatically create a test suite, or augment an existing one, to exercise a GUI-based application with black box, system level test cases. The test cases are synthesized incrementally by employing reinforcement learning. Results indicate improvements over other state of the art techniques in terms of amount of behaviour being exercised by the augmented test suite, and therefore in potential of fault detection. The second paper, “Automatic Test Case Evolution” by Mehdi Mirzaaghaei, Fabrizio Pastore, and Mauro Pezzè, presents a framework to repair test cases that have become obsolete due to evolution and also use information in an existing test suite to generate new test cases. The authors refer to this process as the evolution of a test suite. Experimental evaluation on five different case study systems provides very encouraging results, with a high effectiveness at repairing broken test cases. The third paper, “An Efficient Regression Testing Approach for PHP Web Applications: A Controlled Experiment” by Hyunsook Do and Md. Hossain, introduces a regression testing technique for PHP web applications based on impact analysis and program slicing. A controlled experiment on five non-trivial web applications taken from SourceForge shows a promising reduction of regression testing costs. The special issue editors would like to express their gratitude to the many people who contributed to the successful organization of ICST 2012. Although it is not possible to list them all, a mention goes to the General Chair, Giulio Antoniol, the Industrial Track Chairs, Thomas Ostrand, Saurabh Sinha, and Peter Zimmerer, and the ICST Steering Committee members. The high quality of the papers in this special issue is certainly due to all ICST authors, who dedicated effort and energy in sharing their results with the community, and made up a great pool from which invitations to submit to the journal could be issued; it is due as well to the ICST 2012 Program Committee and the STVR additional reviewers, who provided extensive competent feedback. Last but not least, the STVR chief editors, Robert Hierons and Jeff Offutt, provided expert guidance and important advice throughout the process. Enjoy!

Design by contract (DbC) is a software development methodology that focuses on clearly defining the interfaces between components to produce better quality object-oriented software. Though there exists ample support for DbC for sequential programs, applying DbC to concurrent programs presents several challenges. Using Java as the target programming language, we tackle such challenges by augmenting the Java Modelling Language (JML) and modifying the JML compiler (jmlc) to generate runtime assertion checking code to support DbC in concurrent programs. We applied our solution in a carefully designed case study on a highly concurrent industrial software system from the telecommunications domain to assess the effectiveness of contracts as test oracles in detecting and diagnosing functional faults in concurrent software. Based on these results, clear and objective requirements are defined for contracts to be effective test oracles for concurrent programs whilst balancing the effort to design them. Effort is measured indirectly through the contract complexity measure (CCM), a measure we define. Main results include that contracts of a realistic level of completeness and complexity can detect around 76 percent of faults and reduce the diagnosis effort for such faults tenfold. We, therefore, show that DbC can be applied to concurrent software and can be a valuable tool to improve the economics of software engineering.

This paper discusses a step towards reverse engineering source code to produce UML sequence diagrams, with the aim to aid program comprehension and other activities (e.g., verification). Specifically, our objective being to obtain a lightweight instrumentation and therefore disturb the software behaviour as little as possible in order to eventually produce accurate sequence diagrams. To achieve this, we combine static and dynamic analyses of a Java software, reducing information we collect at runtime (lightweight instrumentation) and compensating for the reduced runtime information with information obtained statically from source code. Static and dynamic information are represented as models and UML diagram generation becomes a model transformation problem. Our validation against a previous, correct approach shows that we indeed reduce the execution overhead inherent to dynamic analysis, while still producing useful diagrams.

This paper presents an industrial case study of coverage-based prioritisation techniques on a real world system with real regression faults. The study evaluates four common and different test case prioritisation techniques and examines the effects of using various coverage criteria on the fault detection rates of the prioritised test suites. The results show that prioritisation techniques that are based on additional coverage with finer grained coverage criteria perform significantly better in fault detection rates. The study also reveals that using modification information does not significantly enhance fault detection rates.

Use case modeling, including use case diagrams and use case specifications (UCSs), is commonly applied to structure and document requirements. UCSs are usually structured but unrestricted textual documents complying with a certain use case template. However, because Use Case Models (UCMods) remain essentially textual, ambiguity is inevitably introduced. In this article, we propose a use case modeling approach, called Restricted Use Case Modeling (RUCM), which is composed of a set of well-defined restriction rules and a modified use case template. The goal is two-fold: (1) restrict the way users can document UCSs in order to reduce ambiguity and (2) facilitate the manual derivation of initial analysis models which, when using the Unified Modeling Language (UML), are typically composed of class diagrams, sequence diagrams, and possibly other types of diagrams. Though the proposed restriction rules and template are based on a clear rationale, two main questions need to be investigated. First, do users find them too restrictive or impractical in certain situations? In other words, can users express the same requirements with RUCM as with unrestricted use cases? Second, do the rules and template have a positive, significant impact on the quality of the constructed analysis models? To investigate these questions, we performed and report on two controlled experiments, which evaluate the restriction rules and use case template in terms of (1) whether they are easy to apply while developing UCMods and facilitate the understanding of UCSs, and (2) whether they help users manually derive higher quality analysis models than what can be generated when they are not used, in terms of correctness, completeness, and redundancy. This article reports on the first controlled experiments that evaluate the applicability of restriction rules on use case modeling and their impact on the quality of analysis models. The measures we have defined to characterize restriction rules and the quality of analysis class and sequence diagrams can be reused to perform similar experiments in the future, either with RUCM or other approaches. Results show that the restriction rules are overall easy to apply and that RUCM results into significant improvements over traditional approaches (i.e., with standard templates, without restrictions) in terms of class correctness and class diagram completeness, message correctness and sequence diagram completeness, and understandability of UCSs.

An Intrusion Detection System (IDS) protects computer networks against attacks and intrusions in combination with firewalls and anti-virus systems. One class of IDS is called signature-based network IDSs as they monitor network traffic, looking for evidence of malicious behaviour as specified in attack descriptions (referred to as signatures). It is common knowledge in the research community that IDSs have problems accurately identifying attacks. In this paper we discuss this accuracy problem and decompose it into a detection problem and a confirmation problem. We then map the evaluation of this accuracy problem to the traditional software verification and validation problem, which allows us to analyze the techniques academics have been using to evaluate their IDS technologies. As a result, we are able to identify areas where research is needed to improve the assessment of the IDS accuracy problem through verification and validation techniques.

Concurrency problems such as starvation and deadlocks should be identified early in the design process. As larger, more complex concurrent systems are being developed, this is made increasingly difficult. We propose here a general approach based on the analysis of specialized design models expressed in the Unified Modeling Language (UML) that uses a specifically designed genetic algorithm to detect concurrency problems. Though the current paper addresses deadlocks and starvation, we will show how the approach can be easily tailored to other concurrency issues. Our main motivations are 1) to devise solutions that are applicable in the context of the UML design of concurrent systems without requiring additional modeling and 2) to use a search technique to achieve scalable automation in terms of concurrency problem detection. To achieve the first objective, we show how all relevant concurrency information is extracted from systems' UML models that comply with the UML Modeling and Analysis of Real-Time and Embedded Systems (MARTE) profile. For the second objective, a tailored genetic algorithm is used to search for execution sequences exhibiting deadlock or starvation problems. Scalability in terms of problem detection is achieved by showing that the detection rates of our approach are, in general, high and are not strongly affected by large increases in the size of complex search spaces.

An Intrusion Detection System (IDS) protects computer networks against attacks and intrusions, in combination with firewalls and anti-virus systems. One class of IDS is called signature-based network IDSs, as they monitor network traffic, looking for evidence of malicious behaviour as specified in attack descriptions (referred to as signatures).Many studies report that IDSs, including signature-based network IDSs, have problems to accurately identify attacks. One possible reason that we observed in our past work, and that is worth investigating further, is that several signatures (i.e., several alarms) can be triggered on the same group of packets, a situation we coined overlapping signatures. This paper presents a technique to precisely and systemat ically quantify the signature overlapping problem of an IDS signature database. The solution we describe is based on set theory and finite state automaton theory, and we experiment with our technique on one widely-used and maintained IDS. Results show that our approach is effective at systematically quantifying the overlap problem in one IDS signature database, and can be potentially used on other IDSs.

The idea behind Design by Contract (DbC) is that a method defines a contract stating the requirements a client needs to fulfill to use it, the precondition, and the properties it ensures after its execution, the post condition. Though there exists ample support for DbC for sequential programs, applying DbC to concurrent programs presents several challenges. We have proposed a solution to these challenges in the context of Java as programming language and the Java Modeling language as specification language. This paper presents our findings when applying our DbC technique on an industrial case study to evaluate the ability of contract-based, runtime assertion checking code at detecting and diagnosing race conditions and deadlocks during system testing. The case study is a highly concurrent industrial system from the telecommunications domain, with actual faults. It is the first work to systematically investigate the impact of contract assertions for the detection of race conditions and deadlocks, along with functional properties, in an industrial system.

The Unified Modeling Language (UML) is an object-oriented analysis and design language widely used to created artifacts during the software system lifecycle. UML being a standard notation, without specific guidelines as to how to use it, it must be applied in the context of a specific software development process. The Unified Process (UP) is one such process, extensively used by the object-oriented community, which delivers software best practices via guidelines for all software lifecycle activities. The UP suggests many artifacts to be produced during the software lifecycle. But many practitioners are reluctant to use those artifacts as they question their benefits. System Sequence Diagrams and System Operation Contracts are artifacts, suggested by Larman in his well-known methodology, to complement standard UP artifacts with the intent of better understanding the input and output events related to the system being designed. This paper presents the results of controlled experiments that investigate the impact of using these artifacts during software development. One way to do that is to study the extent to which those artifacts improve the quality of the Domain Model or reduce the effort necessary to complete this Domain Model. Results show that the use of those artifacts mildly improves the quality of the Domain Model, as long as sufficient training is provided. On the other hand, there is no noticeable evidence that those two artifacts reduce the time to produce the Domain Model.

Though there exists ample support for Design by Contract (DbC) for sequential programs, applying DbC to concurrent programs presents several challenges. In previous work, we extended the Java Modeling Language (JML) with constructs to specify concurrent contracts for Java programs. We present a runtime assertion checker (RAC) for the expanded JML capable of verifying assertions for concurrent Java programs. We systematically evaluate the validity of system testing results obtained via runtime assertion checking using actual concurrent and functional faults on a highly concurrent industrial system from the telecommunications domain.

Model transformation is one of the basic principles of Model Driven Architecture. To build a software system, a sequence of transformations is performed, starting from requirements and ending with implementation. However, requirements are mostly in the form of text, but not a model that can be easily understood by computers; therefore, automated transformations from requirements to analysis models are not easy to achieve. The overall objective of this systematic review is to examine existing literature works that transform textual requirements into analysis models, highlight open issues, and provide suggestions on potential directions of future research. The systematic review led to the analysis of 20 primary studies (16 approaches) obtained after a carefully designed procedure for selecting papers published in journals and conferences from 1996 to 2008 and Software Engineering textbooks. A conceptual framework is designed to provide common concepts and terminology and to define a unified transformation process. This facilitates the comparison and evaluation of the reviewed papers.

A large number of research works have addressed the importance of models in software engineering. However, the adoption of model-based techniques in software organizations is limited since these models are perceived to be expensive and not necessarily cost-effective. Focusing on model-based testing, this paper reports on a series of controlled experiments. It investigates the impact of state machine testing on fault detection in class clusters and its cost when compared with structural testing. Based on previous work showing this is a good compromise in terms of cost and effectiveness, this paper focuses on a specific state-based technique: the round-trip paths coverage criterion. Round-trip paths testing is compared to structural testing, and it is investigated whether they are complementary. Results show that even when a state machine models the behavior of the cluster under test as accurately as possible, no significant difference between the fault detection effectiveness of the two test strategies is observed, while the two test strategies are significantly more effective when combined by augmenting state machine testing with structural testing. A qualitative analysis also investigates the reasons why test techniques do not detect certain faults and how the cost of state machine testing can be brought down.

A number of techniques have been proposed for state-based testing. One well-known technique (criterion) is to traverse the graph representing the state machine and generate a so-called transition tree, in an attempt to exercise round trip paths, i.e., paths that start and end in the same state without any other repeating state. Several hypotheses are made when one uses this criterion: exercising paths in the tree, which do not always trigger complete round trip paths, is equivalent to covering round-trip paths; different traversal algorithms are equivalent. In this paper we investigate whether these assumptions hold in practice, and if they do not, we investigate their consequences. Results also lead us to propose a new transition tree construction algorithm that results in higher efficiency and lower cost.

Abstract A number of coverage criteria have been proposed for testing classes and class clusters modeled with state machines. Previous research has revealed their limitations in terms of their capability to detect faults. As these criteria can be considered to execute the control flow structure of the state machine, we are investigating how data flow information can be used to improve them in the context of UML state machines. More specifically, we investigate how such data flow analysis can be used to further refine the selection of a cost‐effective test suite among alternative, adequate test suites for a given state machine criterion. This paper presents a comprehensive methodology to perform data flow analysis of UML state machines—with a specific focus on identifying the data flow from OCL guard conditions and operation contracts—and applies it to a widely referenced coverage criterion, the round‐trip path (transition tree) criterion. It reports on two case studies whose results show that data flow information can be used to select the best transition tree, in terms of cost effectiveness, when more than one satisfies the transition tree criterion. The results also suggest that different trees are complementary in terms of the data flow that they exercise, thus, leading to the detection of intersecting but distinct subsets of faults. Copyright © 2009 John Wiley & Sons, Ltd.

In the context of object-oriented analysis and design (OOAD), class responsibility assignment is not an easy skill to acquire. Though there are many methodologies for assigning responsibilities to classes, they all rely on human judgment and decision making. Our objective is to provide decision-making support to reassign methods and attributes to classes in a class diagram. Our solution is based on a multi-objective genetic algorithm (MOGA) and uses class coupling and cohesion measurement for defining fitness functions. Our MOGA takes as input a class diagram to be optimized and suggests possible improvements to it. The choice of a MOGA stems from the fact that there are typically many evaluation criteria that cannot be easily combined into one objective, and several alternative solutions are acceptable for a given OO domain model. Using a carefully selected case study, this paper investigates the application of our proposed MOGA to the class responsibility assignment problem, in the context of object-oriented analysis and domain class models. Our results suggest that the MOGA can help correct suboptimal class responsibility assignment decisions and perform far better than simpler alternative heuristics such as hill climbing and a single-objective GA.

An Intrusion Detection System (IDS) is a crucial element of a network security posture. One class of IDS, called signature-based network IDSs, monitors network traffic, looking for evidence of malicious behavior as specified in attack descriptions (referred to as signatures). Many studies have reported that IDSs can generate thousands of alarms a day, many of which are false alarms. The problem often lies in the low accuracy of IDS signatures. It is therefore important to have more accurate signatures in order to reduce the number of false alarms. One part of the false alarm problem is the inability of IDSs to verify attacks (i.e. distinguish between successful and failed attacks). If IDSs were able to accurately verify attacks, this would reduce the number of false alarms a network administrator has to investigate. In this paper, we demonstrate the feasibility of using a data mining algorithm to automatically generate IDS verification rules. We show that this automated approach is effective in reducing the number of false alarms when compared to other widely used and maintained IDSs.

Design by contract (DbC) is a software development methodology that makes use of assertions to produce better quality object-oriented software. The idea behind DbC is that a method defines a contract stating the requirements a client needs to fulfill to use it, the precondition, and the properties it ensures after its execution, the postcondition. Though there exists ample support for DbC for sequential programs, applying DbC to concurrent programs presents several challenges. The first challenge is interference, the product of multiple threads of execution modifying and accessing shared data. The second is the specification of thread-safety properties in the presence of inheritance.We present a solution to these challenges in the context of Java programs by extending the Java modeling language (JML) specification language. We experiment our solution on a large size industrial software system.

Although models have been proven to be helpful in a number of software engineering activities there is still significant resistance to model-driven development. This paper investigates one specific aspect of this larger problem. It addresses the impact of using statecharts for testing class clusters that exhibit a state-dependent behavior. More precisely, it reports on a controlled experiment that investigates their impact on testing fault-detection effectiveness. Code-based, structural testing is compared to statechart-based testing and their combination is investigated to determine whether they are complementary. Results show that there is no significant difference between the fault detection effectiveness of the two test strategies but that they are significantly more effective when combined. This implies that a cost-effective strategy would specify statechart-based test cases early on, execute them once the source code is available, and then complete them with test cases based on code coverage analysis.

Class responsibility assignment is not an easy skill to acquire. Though there are many methodologies for assigning responsibilities to classes, they all rely on human judgment and decision making. Our objective is to provide decision-making help to re-assign methods and attributes to classes in a class diagram. Our solution is based on a multi-objective genetic algorithm (MOGA) and uses class coupling and cohesion measurement. Our MOGA takes as input a class diagram to be optimized and suggests possible improvements to it. The choice of a MOGA stems from the fact that there are typically many evaluation criteria that cannot be easily combined into one objective, and several alternative solutions are acceptable for a given OO domain model. This article presents our approach in detail, our decisions regarding the multi-objective genetic algorithm, and reports on a case study. Our results suggest that the MOGA can help correct suboptimal class responsibility assignment decisions.

Using a specific machine learning technique, this paper proposes a way to identify suspicious statements during debugging. The technique is based on principles similar to Tarantula but addresses its main flaw: its difficulty to deal with the presence of multiple faults as it assumes that failing test cases execute the same fault(s). The improvement we present in this paper results from the use of C4.5 decision trees to identify various failure conditions based on information regarding the test cases' inputs and outputs. Failing test cases executing under similar conditions are then assumed to fail due to the same fault(s). Statements are then considered suspicious if they are covered by a large proportion of failing test cases that execute under similar conditions. We report on a case study that demonstrates improvement over the original Tarantula technique in terms of statement ranking. Another contribution of this paper is to show that failure conditions as modeled by a C4.5 decision tree accurately predict failures and can therefore be used as well to help debugging.

An intrusion detection system (IDS) is a crucial element of a network security posture. Although there are many IDS products available, it is rather difficult to find information about their accuracy. Only a few organizations evaluate these products. Furthermore, the data used to test and evaluate these IDS is usually proprietary. Thus, the research community cannot easily evaluate the next generation of IDS. Toward this end, DARPA provided in 1998, 1999 and 2000 an intrusion detection evaluation data set. However, no new data set has been released by DARPA since 2000, in part because of the cumbersomeness of the task. In this paper, we propose a strategy to address certain aspects of generating a publicly available documented data set for testing and evaluating intrusion detection systems. We also present a tool that automatically analyzes and evaluates IDS using our proposed data set

One of the characteristics of object-oriented software is the complex dependencies that may exist between classes due to generalization and client-server relationships. Hence, where to start testing and how to define an integration strategy are issues that require investigation. A number of techniques exist to order the test of classes with the aim of reducing costs. They usually generate a class test order that is a directed graph indicating in which order classes have to be tested, or which classes can be tested in parallel. This paper shows how such a class test order can be beneficial to testers and designers when conducting various activities, such as planning and scheduling class testing. This is illustrated by means of a case study

Commercial-off-the-Shelf (COTS) components provide a means to construct software (component-based) systems in reduced time and cost. In a COTS component software market there exist component vendors (original developers of the component) and component users (developers of the component-based systems). The former provide the component to the user without source code or design documentation, and as a result it is difficult for the latter to adequately test the component when deployed in their system. In this article we propose a framework that clarifies the roles and responsibilities of both parties so that the user can adequately test the component in a deployment environment and the vendor does not need to release proprietary details. Then, based on this framework we combine and adapt two specification-based testing techniques and describe (and implement) a method for the automated generation of adequate test sets. An evaluation of our approach on a case study demonstrates that it is possible to automatically generate cost effective test sequences and that these test sequences are effective at detecting complex errors.

A stress test methodology aimed at increasing chances of discovering faults related to network traffic in distributed systems is presented. The technique uses the UML 2.0 model of the distributed system under test, augmented with timing information, and is based on an analysis of the control flow in sequence diagrams. It yields stress test requirements that are made of specific control flow paths along with time values indicating when to trigger them. Different variants of our stress testing technique already exist (they stress different aspects of a distributed system) and we focus here on one variant that is designed to identify and to stress test the system at the instant when data traffic on a network is maximal. Using a real-world distributed system specification, we design and implement a prototype distributed system and describe, for that particular system, how the stress test cases are derived and executed using our methodology. The stress test results indicate that the technique is significantly more effective at detecting network traffic-related faults when compared to test cases based on an operational profile.

Software design patterns are documented best practice solutions that can be applied to recurring problems. Although well documented, there are often opportunities to apply them which are overlooked by software designers. This can be the result of inexperience, the sheer complexity of the system, or the fact that design patterns do not always constitute intuitive designs. In this paper, we present a structured methodology for semi-automating the detection of areas within a UML design of a software system that are good candidates for the use of design patterns. This is achieved by the definition of detection rules formalized using the OCL and using a decision tree model. The approach is illustrated on an example GoF design pattern. A prototype tool was developed to show the feasibility of the approach in practical situations, and is used on a case study, producing encouraging results

The empirical assessment of test techniques plays an important role in software testing research. One common practice is to seed faults in subject software, either manually or by using a program that generates all possible mutants based on a set of mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults, thus facilitating the statistical analysis of fault detection effectiveness of test suites; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. Focusing on four common control and data flow criteria (block, decision, C-use, and P-use), this paper investigates this important issue based on a middle size industrial program with a comprehensive pool of test cases and known faults. Based on the data available thus far, the results are very consistent across the investigated criteria as they show that the use of mutation operators is yielding trustworthy results: generated mutants can be used to predict the detection effectiveness of real faults. Applying such a mutation analysis, we then investigate the relative cost and effectiveness of the above-mentioned criteria by revisiting fundamental questions regarding the relationships between fault detection, test suite size, and control/data flow coverage. Although such questions have been partially investigated in previous studies, we can use a large number of mutants, which helps decrease the impact of random variation in our analysis and allows us to use a different analysis approach. Our results are then; compared with published studies, plausible reasons for the differences are provided, and the research leads us to suggest a way to tune the mutation analysis process to possible differences in fault detection probabilities in a specific environment

The Unified Modeling Language (UML) is becoming the de facto standard for software analysis and design modeling. However, there is still significant resistance to model-driven development in many software organizations because it is perceived to be expensive and not necessarily cost-effective. Hence, it is important to investigate the benefits obtained from modeling. As a first step in this direction, this paper reports on controlled experiments, spanning two locations, that investigate the impact of UML documentation on software maintenance. Results show that, for complex tasks and past a certain learning curve, the availability of UML documentation may result in significant improvements in the functional correctness of changes as well as the quality of their design. However, there does not seem to be any saving of time. For simpler tasks, the time needed to update the UML documentation may be substantial compared with the potential benefits, thus motivating the need for UML tools with better support for software maintenance

This paper proposes a methodology and instrumentation infrastructure toward the reverse engineering of UML (Unified Modeling Language) sequence diagrams from dynamic analysis. One motivation is, of course, to help people understand the behavior of systems with no (complete) documentation. However, such reverse-engineered dynamic models can also be used for quality assurance purposes. They can, for example, be compared with design sequence diagrams and the conformance of the implementation to the design can thus be verified. Furthermore, discrepancies can also suggest failures in meeting the specifications. Due to size constraints, this paper focuses on the distribution aspects of the methodology we propose. We formally define our approach using metamodels and consistency rules. The instrumentation is based on aspect-oriented programming in order to alleviate the effort overhead usually associated with source code instrumentation. A case study is discussed to demonstrate the applicability of the approach on a concrete example

Many approaches exist to decide the order in which classes should be integrated during (integration) testing. Most of them, based on an analysis of class dependencies (for instance described in a UML class diagram) aim at producing a partial order indicating which classes should be tested in sequence and which ones can be tested in parallel. We argue in this article that, thanks to the specifics of such a class test order, it is possible to define an incremental strategy for testing classes that promotes reuse during testing, not only along class inheritance hierarchies.

Reactive real-time systems have to react to external events within time constraints: Triggered tasks must execute within deadlines. The goal of this article is to automate, based on the system task architecture, the derivation of test cases that maximize the chances of critical deadline misses within the system. We refer to that testing activity as stress testing. We have developed a method based on genetic algorithms and implemented it in a tool. Case studies were run and results show that the tool may actually help testers identify test cases that will likely stress the system to such an extent that some tasks may miss deadlines.

The empirical assessment of test techniques plays an important role in software testing research. One common practice is to instrument faults, either manually or by using mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. This paper investigates this important question based on a number of programs with comprehensive pools of test cases and known faults. It is concluded that, based on the data available thus far, the use of mutation operators is yielding trustworthy results (generated mutants are similar to real faults). Mutants appear however to be different from hand-seeded faults that seem to be harder to detect than real faults.

In this paper we report on how aspect-oriented programming (AOP), using AspectJ, can be employed to automatically and efficiently instrument contracts and invariants in Java. The paper focuses on the templates to instrument preconditions, postconditions, and class invariants, and the necessary instrumentation for compliance-checking to the Liskov substitution principle.

This article addresses the generation of traces to monitor the execution of distributed Java systems, and investigates the use of aspect-oriented programming (AOP) as the instrumentation strategy to get the necessary information at runtime. The overall objective is to gather enough information to help people understand program executions by abstracting out design details related to thread and distributed communications, for instance under the form of UML sequence diagrams. We show how AspectJ, the main Java implementation of AOP, can be used to solve such issues, assuming RMI is the distribution middleware and thread communications employ specific data structures. The most important aspects are discussed and experiments on a case study are reported.

Empirical studies have shown there is wide variation in cost (e.g., of devising and executing test cases) and effectiveness (at finding faults) across existing state-based coverage criteria. As these criteria can be considered as executing the control flow structure of the statechart, we are attempting to investigate how data flow information can be used to improve their cost-effectiveness. This article presents a comprehensive methodology to perform data flow analysis of UML statecharts, applies it to the round-trip path (transition tree) coverage criterion and reports on two case studies. The results of the case studies show that dataflow information can be used to select the best cost-effective transition tree when more than one satisfies the transition tree criterion. We further propose a more optimal strategy for the transition tree criterion, in terms of cost and effectiveness. The improved tree strategy is evaluated through the two case studies and the results suggest that it is a cost-effective strategy that would fit into many practical situations

The object constraint language (OCL) was introduced as part of the Unified Modeling Language (UML). Its main purpose is to make UML models more precise and unambiguous by providing a constraint language describing constraints that the UML diagrams alone do not convey, including class invariants, operation contracts, and statechart guard conditions. There is an ongoing debate regarding the usefulness of using OCL in UML-based development, questioning whether the additional effort and formality is worth the benefit. It is argued that natural language may be sufficient, and using OCL may not bring any tangible benefits. This debate is in fact similar to the discussion about the effectiveness of formal methods in software engineering, but in a much more specific context. This paper presents the results of two controlled experiments that investigate the impact of using OCL on three software engineering activities using UML analysis models: detection of model defects through inspections, comprehension of the system logic and functionality, and impact analysis of changes. The results show that, once past an initial learning curve, significant benefits can be obtained by using OCL in combination with UML analysis diagrams to form a precise UML analysis model. But, this result is however conditioned on providing substantial, thorough training to the experiment participants.

A number of testing strategies have been proposed using state machines and statecharts as test models in order to derive test sequences and validate classes or class clusters. Though such criteria have the advantage of being systematic, little is known on how cost effective they are and how they compare to each other. This article presents a precise simulation and analysis procedure to analyze the cost-effectiveness of statechart-based testing techniques. We then investigate, using this procedure, the cost and fault detection effectiveness of adequate test sets for the most referenced coverage criteria for statecharts on three different representative case studies. Through the analysis of common results and differences across studies, we attempt to draw more general conclusions regarding the costs and benefits of using the criteria under investigation.

The object constraint language (OCL) was introduced as part of the Unified Modeling Language (UML). Its main purpose is to make UML models more precise by providing a constraint language. For example, operation contracts and statechart guard conditions can be precisely defined using OCL. There has been an ongoing debate on the usefulness of using OCL in UML-based development, questioning whether the additional effort and formality were worth the benefit. This work presents the results of a controlled experiment that investigates the impact of using OCL on model comprehension and maintainability. Current results show that, once past an initial learning curve, significant benefits can be obtained by using OCL in combination with UML diagrams.

This work describes an empirical investigation of the cost effectiveness of well-known state-based testing techniques for classes or clusters of classes that exhibit a state-dependent behavior. This is practically relevant as many object-oriented methodologies recommend modeling such components with statecharts which can then be used as a basis for testing. Our results, based on a series of three experiments, show that in most cases state-based techniques are not likely to be sufficient by themselves to catch most of the faults present in the code. Though useful, they need to be complemented with black-box, functional testing. We focus here on a particular technique, Category Partition, as this is the most commonly used and referenced black-box, functional testing technique. Two different oracle strategies have been applied for checking the success of test cases. One is a very precise oracle checking the concrete state of objects whereas the other one is based on the notion of state invariant (abstract states). Results show that there is a significant difference between them, both in terms of fault detection and cost. This is therefore an important choice to make that should be driven by the characteristics of the component to be tested, such as its criticality, complexity, and test budget.

The use of Unified Modeling Language (UML) analysis/design models on large projects leads to a large number of interdependent UML diagrams. As software systems evolve, those diagrams undergo changes to, for instance, correct errors or address changes in the requirements. Those changes can in turn lead to subsequent changes to other elements in the UML diagrams. Impact analysis is then defined as the process of identifying the potential consequences (side-effects) of a change, and estimating what needs to be modified to accomplish a change. In this article, we propose a UML model-based approach to impact analysis that can be applied before any implementation of the changes, thus allowing an early decision-making and change planning process. We first verify that the UML diagrams are consistent (consistency check). Then changes between two different versions of a UML model are identified according to a change taxonomy, and model elements that are directly or indirectly impacted by those changes (i.e., may undergo changes) are determined using formally defined impact analysis rules (written with Object Constraint Language). A measure of distance between a changed element and potentially impacted elements is also proposed to prioritize the results of impact analysis according to their likelihood of occurrence. We also present a prototype tool that provides automated support for our impact analysis strategy, that we then apply on a case study to validate both the implementation and methodology.

This article is a first attempt towards a comprehensive, systematic methodology for class interface testing in the context of client/server relationships. The proposed approach builds on and combines existing techniques. It first consists in selecting a subset of the method sequences defined for the class testing of the client class, based on an analysis of the interactions between the client and the server methods. Coupling information is then used to determine the conditions, i.e., values for parameters and data members, under which the selected client method sequences are to be executed so as to exercise the interaction. The approach is illustrated by means of an abstract example and its cost-effectiveness is evaluated through a case study.

Abstract A number of activities involved in testing software are known to be difficult and time consuming. Among them is the definition and coding of test oracles and the isolation of faults once failures have been detected. Through a thorough and rigorous empirical study, we investigate how the instrumentation of contracts could address both issues. Contracts are known to be a useful technique in specifying the precondition and postcondition of operations and class invariants, thus making the definition of object‐oriented analysis or design elements more precise. It is one of the reasons the Object Constraint Language (OCL) was made part of the Unified Modeling Language. Our aim in this paper is to reuse and instrument contracts to ease testing. A thorough case study is run where we define OCL contracts, instrument them using a commercial tool and assess the benefits and limitations of doing so to support the automated detection of failures and the isolation of faults. As contracts can be defined at various levels of detail, we also investigate the cost and benefit of using contracts at different levels of precision. We then draw practical conclusions regarding the applicability of the approach and its limitations. Copyright © 2003 John Wiley & Sons, Ltd.

The issue of ordering class integration in the context of integration testing has been discussed by a number of researchers. More specifically, strategies have been proposed to generate a test order while minimizing stubbing. Recent papers have addressed the problem of deriving an integration order in the presence of dependency cycles in the class diagram. Such dependencies represent a practical problem as they make any topological ordering of classes impossible. Three main approaches, aimed at "breaking" cycles, have been proposed. The first one was proposed by Tai and Daniels (1999) and is based on assigning a higher-level order according to aggregation and inheritance relationships and a lower-level order according to associations. The second one was proposed by Le Traon et al. (2000) and is based on identifying strongly connected components in the dependency graph. The third one was proposed by Briand et al. (2000); it combines some of the principles of the two previous approaches and addresses some of their shortcomings (e.g., the first approach may result into unnecessary stubbing whereas the second may lead to breaking cycles by "removing" aggregation or inheritance dependencies, thus leading to complex stubbing). This paper reviews these strategies (principles are described, advantages and drawbacks are precisely investigated) and provides both analytical and empirical comparisons based on five case studies.

We present a methodology and a tool to support test selection from regression test suites based on change analysis in object-oriented designs. We assume that designs are represented using the Unified Modeling Language (UML) and we propose a formal mapping between design changes and a classification of regression test cases, i.e., three categories: reusable, retestable, and obsolete. We provide evidence of the feasibility of the methodology and its usefulness by using our prototype tool on an industrial case study.

A number of strategies have been proposed for state-based class testing. An important proposal made by Chow (1978), that was subsequently adapted by Binder (1999), consists in deriving test sequences covering all round-trip paths in a finite state machine (FSMs). Based on a number of (rather strong) assumptions, and for traditional FSMs, it can be demonstrated that all operation and transfer errors in the implementation can be uncovered. Through experimentation, this paper investigates this strategy when used in the context of UML statecharts. Based on a set of mutation operators proposed for object-oriented code we seed a significant number of faults in an implementation of a specific container class. We then investigate the effectiveness of four test teams at uncovering faults, based on the round-trip path strategy, and analyze the faults that seem to be difficult to detect. Our main conclusion is that the round-trip path strategy is reasonably effective at detecting faults (87% average as opposed to 69% for size-equivalent, random test cases) but that a significant number of faults can only exhibit a high detection probability by augmenting the round-trip strategy with a traditional black-box strategy such as category-partition testing. This increases the number of test cases to run -and therefore the cost of testing- and a cost-benefit analysis weighting the increase of testing effort and the likely gain in fault detection is necessary.

We present here an improved strategy to devise optimal integration test orders in object-oriented systems. Our goal is to minimize the complexity of stubbing during integration testing as this has been shown to be a major source of expenditure. Our strategy to do so is based on the combined use of inter-class coupling measurement and genetic algorithms. The former is used to assess the complexity of stubs and the latter is used to minimize complex cost functions based on coupling measurement. Using a precisely defined procedure, we investigate this approach in a case study involving a real system. Results are very encouraging as the approach clearly helps obtaining systematic and optimal results.

The issue of ordering class integration in the context of integration testing of object-oriented software has been discussed by a number of researchers. More specifically, strategies have been proposed to generate a test order while minimizing stubbing. Recent papers have addressed the problem of deriving an integration order in the presence of dependency cycles in the class diagram. Such dependencies represent a practical problem as they make any topological ordering of classes impossible. The paper proposes a strategy that integrates two existing methods aimed at "breaking" cycles so as to allow a topological order of classes. The first one was proposed by K.-C. Tai and F.J. Daniels (1999) and is based on assigning a higher-level order according to aggregation and inheritance relationships and a lower-level order according to associations. The second one was proposed by Y. Le Traon et al. (2000) and is based on identifying strongly connected components in the dependency graph. Among other things, the former approach may result in unnecessary stubbing whereas the latter may lead to breaking cycles by "removing" aggregation or inheritance dependencies, thus leading to complex stubbing. We propose an approach that combines some of the principles of both approaches and addresses some of their shortcomings. All approaches (principles, benefits, drawbacks) are thoroughly compared by the means of a case study, based on a real system written in Java.

One of the characteristics of object-oriented software is the complex dependency that may exist between classes due to inheritance, association and aggregation relationships. Hence, where to start testing and how to define an integration strategy are issues that require further investigation. This paper presents an approach to define a test order by exploiting a model produced during design stages (e.g., using OMT, UML), namely the class diagram. Our goal is to minimize the number of stubs to be constructed in order to decrease the cost of testing. This is done by testing a class after the classes it depends on. The novelty of the test order lies in the fact that it takes account of: (i) dynamic (polymorphism) dependencies; (ii) abstract classes that cannot be instantiated, making some testing levels infeasible. The test order is represented by a graph showing which testing levels must be done in sequence and which ones may be done independently. It also provides information about the classes involved in each level and how they are involved (e.g., instantiation or not). The approach is implemented in a tool called TOONS (Testing level generator for Object-OrieNted Software). It is applied to an industrial case study from the avionics domain.