To assess the quality of web application test cases, web developers currently measure code coverage. Although code coverage has traditionally been a popular test adequacy criterion, we believe it alone is not adequate for assessing the quality of web application test cases. We propose a set of novel DOM-based test adequacy criteria for web applications. These criteria aim at measuring coverage at two granularity levels, (1) the percentage of DOM states and transitions covered in the total state space of the web application under test, and (2) the percentage of elements covered in each particular DOM state. We present a technique and tool, called DomCovery, which automatically extracts and measures the proposed adequacy criteria and generates a visual DOM coverage report. Our evaluation shows that there is no correlation between code coverage and DOM coverage. A controlled experiment illustrates that participants using DomCovery completed coverage related tasks 22% more accurately and 66% faster.

To test web applications, developers currently write test cases in frameworks such as Selenium. On the other hand, most web test generation techniques rely on a crawler to explore the dynamic states of the application. The first approach requires much manual effort, but benefits from the domain knowledge of the developer writing the test cases. The second one is automated and systematic, but lacks the domain knowledge required to be as effective. We believe combining the two can be advantageous. In this paper, we propose to (1) mine the human knowledge present in the form of input values, event sequences, and assertions, in the human-written test suites, (2) combine that inferred knowledge with the power of automated crawling, and (3) extend the test suite for uncovered/unchecked portions of the web application under test. Our approach is implemented in a tool called Testilizer. An evaluation of our approach indicates that Testilizer (1) outperforms a random test generator, and (2) on average, can generate test suites with improvements of up to 150% in fault detection rate and up to 30% in code coverage, compared to the original test suite.

Bug repository systems have become an integral component of software development activities. Ideally, each bug report should help developers to find and fix a software fault. However, there is a subset of reported bugs that is not (easily) reproducible, on which developers spend considerable amounts of time and effort. We present an empirical analysis of non-reproducible bug reports to characterize their rate, nature, and root causes. We mine one industrial and five open-source bug repositories, resulting in 32K non-reproducible bug reports. We (1) compare properties of non-reproducible reports with their counterparts such as active time and number of authors, (2) investigate their life-cycle patterns, and (3) examine 120 Fixed non-reproducible reports. In addition, we qualitatively classify a set of randomly selected non-reproducible bug reports (1,643) into six common categories. Our results show that, on average, non-reproducible bug reports pertain to 17% of all bug reports, remain active three months longer than their counterparts, can be mainly (45%) classified as "Interbug Dependencies'', and 66% of Fixed non-reproducible reports were indeed reproduced and fixed.

SUMMARY Software systems evolve incrementally both during and after development, and many test cases become obsolete while software evolves. Updating test suites in the context of software evolution is a complex and time consuming activity. This article focuses on the problem of updating test suites automatically, and identifies eight scenarios that allow either to repair test cases or to use test cases to generate new ones, and proposes eight test evolution algorithms that automatically repair and generate test cases by adapting existing ones. This article presents a framework, TestCareAssistant (TCA), that implements the algorithms to support the evolution of test suites written in Java. The framework has been extensively evaluated on five different open source projects where it has been applied to repair 138 broken test cases, and to generate the test cases for 727 new classes and 2462 new methods. The results obtained with TCA indicate that the approach can successfully repair 90% of the broken test cases, create test cases that cover a large amount of code and complement the test cases that can be generated by state of the art techniques. Copyright © 2014 John Wiley & Sons, Ltd.

Software systems evolve during development and maintenance, and many test cases designed for the early versions of the system become obsolete during the software lifecycle. Repairing test cases that do not compile due to changes in the code under test and generating new test cases to test the changed code is an expensive and time consuming activity that could benefit from automated approaches. In this paper we propose an approach for automatically repairing and generating test cases during software evolution. Differently from existing approaches to test case generation, our approach uses information available in existing test cases, defines a set of heuristics to repair test cases invalidated by changes in the software, and generate new test cases for evolved software. The results obtained with a prototype implementation of the technique show that the approach can effectively maintain evolving test suites, and perform well compared to competing approaches.

Software evolves continuously, and developers need to retest it frequently. To save time and effort, developers often reuse existing test cases to verify the functionality of software systems after changes, but they often need to adapt or augment the test cases to match the new characteristics of the software systems. Adapting test cases is tedious and expensive. Current automated techniques often generate invalid and incomplete test cases, and require manual inspection and correction of the generated test cases.

When software systems evolve, for example due to fault fixes, modification of functionalities or refactoring activities, test cases may become obsolete thus generating wrong results or even not being executable or compilable. Maintaining test cases is expensive and time consuming, and often test cases are discarded by software developers due to high maintenance costs. This paper presents TestCareAssistant, a technique that combines data-flow analysis with program diffing for automatically repairing test cases that become obsolete because of changes in method declarations (addition, removal, or type modification of parameters or return values). The paper illustrates the efficacy of TestCareAssistant by analyzing the impact of method declarations changes on the executability of test cases, and by presenting the preliminary results of applying TestCareAssistant to repair 22 test cases.

Conventional software verification and validation (V&V) approaches for real-time and reactive programs are not practical because applying formal methods is too complex and testing methods are incomplete. It seems runtime verification of programs is a novel approach as a compliment to testing methods. We use event-based real-time logic (ERL) as a specification language in order to simply specify safety properties. By applying aspect-oriented approach to instrumentation, we integrate runtime verification module (i.e. Monitor) with program itself and minimize overhead of runtime verification too. Our proposed method, RVERL, consists of three phases. First, safety properties are extracted from program requirements specification. Second, properties are mapped to timing, functional and deadline aspects which constitute the monitor. Then it is weaved to the program source code. Third, at the execution time, the monitor as a wrapper observes program behavior and protects the program against its misbehavior (i.e., property violations). Experiments on rail road crossing problem show that the RVERL reveals all property violations while overhead of runtime verification is just about 4 percent.