Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures.

Software evolves continuously. Its behavior must be validated by engineers who perform daily development and maintenance tasks. However, despite its high importance, information needs for validating evolving software has not been systematically studied in an industrial setting. Such lack of empirical knowledge hinders attempts to understand this fundamental practice and improve the corresponding tool support. This paper presents a study on 194 Site Reliability Engineers (SREs) at Google to explore their information needs. The results suggest several directions where software engineering researchers may consider putting effort to develop new techniques and tool support that matter to practitioners.

This paper presents a technique to detect inadequate (i.e., missing or ambiguous) diagnostic messages for configuration errors issued by a configurable software system. The technique injects configuration errors into the software under test, monitors the software outcomes under the injected configuration errors, and uses natural language processing to analyze the output diagnostic message caused by each configuration error. The technique reports diagnostic messages that may be unhelpful in diagnosing a configuration error. We implemented the technique for Java in a tool, ConfDiagDetector. In an evaluation on 4 real-world, mature configurable systems, ConfDiagDetector reported 43 distinct inadequate diagnostic messages (25 missing and 18 ambiguous). 30 of the detected messages have been confirmed by their developers, and 12 more have been identified as inadequate by users in a user study. On average, Conf- DiagDetector required 5 minutes of programmer time and 3 minutes of compute time to detect each inadequate diagnostic message.

Modern software often exposes configuration options that enable users to customize its behavior. During software evolution, developers may change how the configuration options behave. When upgrading to a new software version, users may need to re-configure the software by changing the values of certain configuration options.

In a test suite, all the test cases should be independent: no test should affect any other test’s result, and running the tests in any order should produce the same test results. Techniques such as test prioritization generally assume that the tests in a suite are independent. Test dependence is a little-studied phenomenon. This paper presents five results related to test dependence.

Debugging and isolating changes responsible for regression test failures are some of the most challenging aspects of modern software development. Automatic bug localization techniques reduce the manual effort developers spend examining code, for example, by focusing attention on the minimal subset of recent changes that results in the test failure, or on changes to components with most dependencies or highest churn. We observe that another subset of changes is worth the developers' attention: the complement of the maximal set of changes that does not produce the failure. While for simple, independent source-code changes, existing techniques localize the failure cause to a small subset of those changes, we find that when changes interact, the failure cause is often in our proposed subset and not in the subset existing techniques identify. In studying 45 regression failures in a large, open-source project, we find that for 87% of those failures, the complement of the maximal passing set of changes is different from the minimal failing set of changes, and that for 78% of the failures, our technique identifies relevant changes ignored by existing work. These preliminary results suggest that combining our ideas with existing techniques, as opposed to using either in isolation, can improve the effectiveness of bug localization tools.

We present a technique that simplifies tests at the semantic level. We first formalize the semantic test simplification problem, and prove it is NP-hard. Then, we propose a heuristic algorithm, SimpleTest, that automatically transforms a test into a simpler test, while still preserving a given property. The key insight of SimpleTest is to reconstruct an executable and simpler test that exhibits the given property from the original one. Our preliminary study on 7 real-world programs showed the usefulness of SimpleTest.

This paper presents ConfDiagnoser, an automated configuration error diagnosis tool for Java software. Conf-Diagnoser identifies the root cause of a configuration error - a single configuration option that can be changed to produce desired behavior. It uses static analysis, dynamic profiling, and statistical analysis to link the undesired behavior to specific configuration options. ConfDiagnoser differs from existing approaches in two key aspects: it does not require users to provide a testing oracle (to check whether the software functions correctly) and thus is fully-automated; and it can diagnose both crashing and non-crashing errors. We demonstrated ConfDiagnoser's accuracy and speed on 5 non-crashing configuration errors and 9 crashing configuration errors from 5 configurable software systems.

The behavior of a software system often depends on how that system is configured. Small configuration errors can lead to hard-to-diagnose undesired behaviors. We present a technique (and its tool implementation, called ConfDiagnoser) to identify the root cause of a configuration error - a single configuration option that can be changed to produce desired behavior. Our technique uses static analysis, dynamic profiling, and statistical analysis to link the undesired behavior to specific configuration options. It differs from existing approaches in two key aspects: it does not require users to provide a testing oracle (to check whether the software functions correctly) and thus is fully automated; and it can diagnose both crashing and non-crashing errors. We evaluated ConfDiagnoser on 5 non-crashing configuration errors and 9 crashing configuration errors from 5 configurable software systems written in Java. On average, the root cause was ConfDiagnoser's fifth-ranked suggestion; in 10 out of 14 errors, the root cause was one of the top 3 suggestions; and more than half of the time, the root cause was the first suggestion.

A workflow is a sequence of UI actions to complete a specific task. In the course of a GUI application's evolution, changes ranging from a simple GUI refactoring to a complete rearchitecture can break an end-user's well-established workflow. It can be challenging to find a replacement workflow. To address this problem, we present a technique (and its tool implementation, called FlowFixer) that repairs a broken workflow. FlowFixer uses dynamic profiling, static analysis, and random testing to suggest a replacement UI action that fixes a broken workflow.

Many computer end-users, such as research scientists and business analysts, need to frequently query a database, yet lack enough programming knowledge to write a correct SQL query. To alleviate this problem, we present a programming by example technique (and its tool implementation, called SQLSynthesizer) to help end-users automate such query tasks. SQLSynthesizer takes from users an example input and output of how the database should be queried, and then synthesizes a SQL query that reproduces the example output from the example input. If the synthesized SQL query is applied to another, potentially larger, database with a similar schema, the synthesized SQL query produces a corresponding result that is similar to the example output. We evaluated SQLSynthesizer on 23 exercises from a classic database textbook and 5 forum questions about writing SQL queries. SQLSynthesizer synthesized correct answers for 15 textbook exercises and all 5 forum questions, and it did so from relatively small examples.

Software change impact analysis (CIA) is a key technique for identifying unpredicted and potential effects caused by changes made to software. Different changes have different ripple effects to other parts in the program, even some changes do not affect other entities in spite of some dependencies existing between these entities and the modified one. This induces imprecision if such a factor is neglected. This article proposes a static CIA technique which considers the impact rules of different change types to predict the change effects. Input of our CIA includes changed classes, class methods and class fields, and the output is composed of potentially affected classes, class methods, and class fields. Precision improvement of the CIA technique relies on three aspects: change types of a modified entity, dependencies between the modified entity and other entities, and a precise initial impact set (IIS), on which the final impact set (FIS) is computed. Experimental case studies demonstrate the effectiveness of our technique, and present its potential applications in software maintenance.

SUMMARY Software change impact analysis (CIA) is a technique for identifying the effects of a change, or estimating what needs to be modified to accomplish a change. Since the 1980s, there have been many investigations on CIA, especially for code‐based CIA techniques. However, there have been very few surveys on this topic. This article tries to fill this gap. And 30 papers that provide empirical evaluation on 23 code‐based CIA techniques are identified. Then, data was synthesized against four research questions. The study presents a comparative framework including seven properties, which characterize the CIA techniques, and identifies key applications of CIA techniques in software maintenance. In addition, the need for further research is also presented in the following areas: evaluating existing CIA techniques and proposing new CIA techniques under the proposed framework, developing more mature tools to support CIA, comparing current CIA techniques empirically with unified metrics and common benchmarks, and applying the CIA more extensively and effectively in the software maintenance phase. Copyright © 2012 John Wiley & Sons, Ltd.

To keep a Graphical User Interface (GUI) responsive and active, a GUI application often has a main UI thread (or event dispatching thread) and spawns separate threads to handle lengthy operations in the background, such as expensive computation, I/O tasks, and network requests. Many GUI frameworks require all GUI objects to be accessed exclusively by the UI thread. If a GUI object is accessed from a non-UI thread, an invalid thread access error occurs and the whole application may abort.

In object-oriented programs, a unit test often consists of a sequence of method calls that create and mutate objects. It is challenging to automatically generate sequences that are legal and behaviorally-diverse, that is, reaching as many different program states as possible.

In an object-oriented program, a unit test often consists of a sequence of method calls that create and mutate objects, then use them as arguments to a method under test. It is challenging to automatically generate sequences that are legal and behaviorally-diverse, that is, reaching as many different program states as possible.

A failed test reveals a potential bug in the tested code. Developers need to understand which parts of the test are relevant to the failure before they start bug-fixing. This paper presents a fully-automated technique (and its tool implementation, called FailureDoc) to explain a failed test. FailureDoc augments the failed test with explanatory documentation in the form of code comments. The comments indicate changes to the test that would cause it to pass, helping programmers understand why the test fails. We evaluated FailureDoc on five real-world programs. FailureDoc generated meaningful comments for most of the failed tests. The inferred comments were concise and revealed important debugging clues. We further conducted a user study. The results showed that FailureDoc is useful in bug diagnosis.

Software change impact analysis (CIA) is a key technique for identifying unpredicted and potential effects caused by changes made to software. Different change types often have different impact mechanisms, even some changes do not impact other entities in programs in spite of some dependences existed between these entities and the modified entity. In this paper, we propose a static CIA technique, which considers different impact mechanisms and rules of different change types, to calculate the impact sets. Precision improvement of the impact sets relies on 3 aspects: change types of a modified entity, dependences between the modified entity and other entities, and the intuition that to win at the start -- if the initial impact set is estimated more accurately, then the final impact set depending on this initial impact set will be more precise. Experimental case study demonstrates the effectiveness of our technique, and its potential applications in software maintenance.

Program call graph representation can be used to support many tasks in compiler optimization, program comprehension, and software maintenance. During software evolution, the call graph needs to remain fairly precise and be updated quickly in response to software changes. In this paper, we present an approach to incremental update, instead of exhaustive analysis of the initially constructed call graph in AspectJ software. Our approach first decomposes the source code edits between the updated and initial software versions into a set of atomic change representations, which capture the semantic differences. Then, we explore the relationship between atomic changes and call graph to incrementally update the initially constructed graph, instead of rebuilding it from the ground up. We implement the reanalysis approach on top of the ajc AspectJ compiler and perform an empirical study on 24 versions of eight AspectJ benchmarks. The experiment result shows that our approach can reduce a large portion of unnecessary reanalysis cost as program changes occur, and significant savings are observed for the incremental reconstruction of AspectJ call graph in comparison with an exhaustive analysis, with no loss in precision.

Change impact analysis is a useful technique for software evolution. It determines the effects of a source editing session and provides valuable feedbacks to the programmers for making correct decisions. Recently, many techniques have been proposed to support change impact analysis of procedural or object-oriented software, but seldom effort has been made for aspect-oriented software. In this paper we propose a new change impact analysis technique for AspectJ programs. At the core of our approach is the atomic change representation which captures the semantic differences between two versions of an AspectJ program. We also present an impact analysis model, based on AspectJ call graph construction, to determine the affected program fragments, affected tests and their responsible changes. The proposed techniques have been implemented in Celadon, a change impact analysis framework for AspectJ programs. We performed an empirical evaluation on 24 versions of eight AspectJ benchmarks. The result shows that our proposed technique can effectively perform change impact analysis and provide valuable information in AspectJ software evolution.

Aspect-oriented programming (AOP) is gaining popularity with the wider adoption of languages such as AspectJ. During AspectJ software evolution, when regression tests fail, it may be tedious for programmers to find out the failure-inducing changes by manually inspecting all code editing. To eliminate the expensive effort spent on debugging, we developed AutoFlow, an automatic debugging tool for AspectJ software. AutoFlow integrates the potential of delta debugging algorithm with the benefit of change impact analysis to narrow down the search for faulty changes. It first uses change impact analysis to identify a subset of responsible changes for a failed test, then ranks these changes according to our proposed heuristic (indicating the likelihood that they may have contributed to the failure), and finally employs an improved delta debugging algorithm to determine a minimal set of faulty changes. The main feature of AutoFlow is that it can automatically reduce a large portion of irrelevant changes in an early phase, and then locate faulty changes effectively.

Aspect-oriented software development (AOSD) is gaining popularity with the wider adoption of languages such as AspectJ. However, though the state-of-the-art aspect-oriented programming environment (such as AJDT in the Eclipse IDE) provides powerful capabilities to check the syntactic or grammar errors in AspectJ programs, it fails to detect potential semantic defects in aspect-oriented software systems. In this paper, we present XFindBugs, an eXtended FindBugs for AspectJ, to help programmers find potential bugs in AspectJ applications through static analysis. XFindBugs supports 17 bug patterns to cover common error-prone features in an aspect-oriented system, and integrates the corresponding bug detectors into the FindBugs framework. We evaluate XFindBugs on a number of large-scale open source AspectJ projects (306,800 LOC in total). In our evaluation, XFindBugs confirms 7 reported bugs and finds 257 previously unknown defects. Our experiment also indicates that the bug patterns supported in XFindBugs exist in real-world software systems, even for mature applications by experienced programmers.

When regression tests fail unexpectedly after a long session of editing, it may be tedious for programmers to find out the failure-inducing changes by manually inspecting all code edits. To eliminate the expensive effort spent on debugging, we present a hybrid approach, which combines both static and dynamic analysis techniques, to automatically identify the faulty changes. Our approach first uses static change impact analysis to isolate a subset of responsible changes for a failed test, then utilizes the dynamic test execution information to rank these changes according to our proposed heuristic (indicating the likelihood that they may have contributed to the failure), and finally employs an improved Three-Phase delta debugging algorithm, working from the coarse method level to the fine statement level, to find a minimal set of faulty statements.

In this paper, we propose a fine-grained coupling metrics suite for aspect-oriented (AO) systems, to measure software changes during system evolution. We also present a correlation model in terms of intermediate processes, for better evaluating the relation between coupling metrics and system maintainability. To investigate the practicability of our proposed model, we have implemented a coupling metrics analysis tool called AJMetrics and performed an empirical study on eight AspectJ benchmarks. The experiment result suggests that our correlation model provides useful information to evaluate the maintainability of AO systems.

Bug patterns are erroneous code idioms or bad coding practices that have been proved fail time and time again. They mainly arise from the misunderstanding of language features, the use of erroneous design patterns or simple mistakes sharing the common behaviors. Aspect-oriented programming (AOP) is a new technique to separate the cross-cutting concerns for improving modularity in software design and implementation. However, there is no effective debugging technique for aspect-oriented programs until now and none of the prior researches focused on the identification of bug patterns in aspect-oriented programs. In this paper, we present six bug patterns in AspectJprogramming language and show the corresponding example for each bug pattern to help to illustrate the symptoms of these patterns. We take this as the first step to provide an underlying basis on testing and debugging of AspectJ programs.