Continuous integration (CI) is a software engineering practice that advocates the frequent integration of software through an automated build process. Existing research has explored the benefits of CI, such as detecting errors earlier in the software life-cycle. Although CI heavily focuses on automated tests, it is still unclear whether CI is associated with better code coverage, which could be a major benefit of using CI. To investigate whether CI is associated with an improvement in code coverage, our work compares 30 projects that adopted CI (CI projects) and 30 projects that have never adopted CI (NOCI projects). In total, we studied 1,440 versions from different projects to analyze trends in code coverage related to CI. While evaluating trends of code coverage within CI and NOCI projects, we observe more projects with rising trends of code coverage in CI projects (50%) than NOCI projects (10%). Moreover, the maintaining trends are different, as CI projects tend to stabilize at a higher code coverage rate than NOCI projects. Investigating the CI projects alone, the statistical evidences indicate that the adoption of CI is associated with the increase in code coverage. The findings of this study, therefore, reveal a positive association between CI and a higher code coverage rate.

The information available in crash reports has been used to understand the root cause of bugs and improve the overall quality of systems. Nonetheless, crash reports often lead to a huge amount of information, being necessary to consolidate the crash report data into groups, according to a set of well-defined criteria. Recent research work have proposed different criteria and techniques to group crash report data, making more effective the process of finding the root causes of a bug and showing the performance of the approaches in the context of open source applications (such as IDEs and web browsers). In spite of that, it is still not clear how these approaches perform in other application domains, such as enterprise systems. In this paper, we present an industrial study in this field. We tailor existing approaches to find and group correlated crash reports, and identify buggy files in the domain of web-based systems. We then evaluate the performance of the resulting criteria and technique in industrial settings - identifying and ranking the classes that are more likely to contribute to a crash and thus might need a fix. We also check if the methods changed by the developers to fix a bug are present in the stack traces of the crash report groups used to identify the buggy classes. Our study provides new pieces of evidence of the potential use of crash report groups to indicate buggy classes and methods using stack traces information. For instance, we successfully identify buggy classes with recall varying from 61.4% to 77.3%, considering the top 1, top 3, top 5, and top 10 suspicious buggy files identified and ranked by our approach. We also found that 80% of changed methods from the closed bug fix issues appeared in related stack traces of the crash report groups. Finally, the approach also received positive response from the project leaders of the evaluated projects to help their bug resolution processes.

Exception handling mechanisms are a common feature of mainstream programming languages to deal with exceptional conditions.If on the one hand they allow the programmer to prepare the code to deal with exceptional conditions, on the other hand they can become a source of bugs that threatens the application robustness -studies have shown that uncaught exceptions are the main cause of application crashes.To keep track of crashes and enable an easier fault localization, several applications, nowadays, use crash reporting tools.In this work, we propose a tool named CrashAwareDev, integrated with the Eclipse IDE, which mines the information available in crash reporting tools to support programmers in their day-to-day activities.The proposed tool alerts developers about the classes related to recent crashes and warns about source code characteristics (bug patterns) that can be related to future crashes (similar to the ones that have happened).Doing so the tool aims at bringing the development environment closer to crash reporting tools, that are usually only used for bug fixing or for extracting robustness metrics.A case study was conducted and showed that the tool can support software development by displaying bug pattern alerts directly in the source code, signaling the classes involved in recent faults, and speeding up the crash's fault localization within the development environment itself.

Despite being an old language feature, Java exception handling code is one of the least understood parts of many systems. Several studies have analyzed the characteristics of exception handling code, trying to identify common practices or even link such practices to software bugs. Few works, however, have investigated exception handling issues from the point of view of developers. None of the works have focused on discovering exception handling guidelines adopted by current systems - which are likely to be a driver of common practices. In this work, we conducted a qualitative study based on semi-structured interviews and a survey whose goal was to investigate the guidelines that are (or should be) followed by developers in their projects. Initially, we conducted semi-structured interviews with seven experienced developers, which were used to inform the design of a survey targeting a broader group of Java developers (i.e., a group of active Java developers from top-starred projects on GitHub). We emailed 863 developers and received 98 valid answers. The study shows that exception handling guidelines usually exist (70%) and are usually implicit and undocumented (54%). Our study identifies 48 exception handling guidelines related to seven different categories. We also investigated how such guidelines are disseminated to the project team and how compliance between code and guidelines is verified; we could observe that according to more than half of respondents the guidelines are both disseminated and verified through code inspection or code review. Our findings provide software development teams with a means to improve exception handling guidelines based on insights from the state of practice of 87 software projects.

The exception handling policy of a system comprises the set of design rules that specify its exception handling behavior (how exceptions should be handled and thrown in a system). Such policy is usually undocumented and implicitly defined by the system architect. Developers are usually unaware of such rules and may think that by just sprinkling the code with catch-blocks they can adequately deal with the exceptional conditions of a system. As a consequence, the exception handling code once designed to make the program more reliable may become a source of faults (e.g., the uncaught exceptions are one of the main causes of crashes in current Java applications). To mitigate such problem, we propose Exception Policy Expert (EPE), a tool embedded in Eclipse IDE that warns developers about policy violations related to the code being edited. A case study performed in a real development context showed that the tool could indeed make the exception handling policy explicit to the developers during development.

The approach proposed by Silwerski, Zimmermann, and Zeller (SZZ) for identifying bug-introducing changes is at the foundation of several research areas within the software engineering discipline. Despite the foundational role of SZZ, little effort has been made to evaluate its results. Such an evaluation is a challenging task because the ground truth is not readily available. By acknowledging such challenges, we propose a framework to evaluate the results of alternative SZZ implementations. The framework evaluates the following criteria: (1) the earliest bug appearance, (2) the future impact of changes, and (3) the realism of bug introduction. We use the proposed framework to evaluate five SZZ implementations using data from ten open source projects. We find that previously proposed improvements to SZZ tend to inflate the number of incorrectly identified bug-introducing changes. We also find that a single bug-introducing change may be blamed for introducing hundreds of future bugs. Furthermore, we find that SZZ implementations report that at least 46 percent of the bugs are caused by bug-introducing changes that are years apart from one another. Such results suggest that current SZZ implementations still lack mechanisms to accurately identify bug-introducing changes. Our proposed framework provides a systematic mean for evaluating the data that is generated by a given SZZ implementation.

This paper presents an empirical study whose goal was to investigate the exception handling strategies adopted by Java libraries and their potential impact on the client applications. In this study, exception flow analysis was used in combination with manual inspections in order: (i) to characterize the exception handling strategies of existing Java libraries from the perspective of their users; and (ii) to identify exception handling anti-patterns. We extended an existing static analysis tool to reason about exception flows and handler actions of 656 Java libraries selected from 145 categories in the Maven Central Repository. The study findings suggest a current trend of a high number of undocumented API runtime exceptions (i.e., @throws in Javadoc) and Unintended Handler problem. Moreover, we could also identify a considerable number of occurrences of exception handling anti-patterns (e.g. Catch and Ignore). Finally, we have also analyzed 647 bug issues of the 7 most popular libraries and identified that 20.71% of the reports are defects related to the problems of the exception strategies and anti-patterns identified in our study. The results of this study point to the need of tools to better understand and document the exception handling behavior of libraries.

One of the tasks of reviewers is to verify that code modifications are well tested. However, current tools offer little support in understanding precisely how changes to the code relate to changes to the tests. In particular, it is hard to see whether (modified) test code covers the changed code. To mitigate this problem, we developed Operias, a tool that provides a combined visualization of fine-grained source code differences and coverage impact. Operias works both as a stand-alone tool on specific project versions and as a service hooked to GitHub. In the latter case, it provides automated reports for each new pull request, which reviewers can use to assess the code contribution. Operias works for any Java project that works with maven and its standard Cobertura coverage plugin. We present how Operias could be used to identify test-related problems in real-world pull requests. Operias is open source and available on GitHub with a demo video: https://github.com/SERG-Delft/operias

This paper reports on a study mining the exception stack traces included in 159,048 issues reported on Android projects hosted in GitHub (482 projects) and Google Code (157 projects). The goal of this study is to investigate whether stack trace information can reveal bug hazards related to exception handling code that may lead to a decrease in application robustness. Overall 6,005 exception stack traces were extracted, and subjected to source code and byte code analysis. The out-comes of this study include the identification of the following bug hazards: (i) unexpected cross-type exception wrappings (for instance, trying to handle an instance of Out Of Memory Error "hidden" in a checked exception) which can make the exception-related code more complex and negatively impact the application robustness, (ii) undocumented runtime exceptions thrown by both the Android platform and third party libraries, and (iii) undocumented checked exceptions thrown by the Android Platform. Such undocumented exceptions make it difficult, and most of the times infeasible for the client code to protect against "unforeseen" situations that may happen while calling third-party code. This study provides further insights on such bug hazards and the robustness threats they impose to Android apps as well as to other systems based on the Java exception model.

Exception handling (EH) is a well-known mechanism that aims at improving software reliability in a modular way - allowing a better separation between the code that deals with exceptional conditions and the code that deals with the normal control flow of a program. Although the exception handling mechanism was conceived almost 40 years ago, formulating a reasonable design of exception handling code is still considered a challenge, which might hinder its widespread use. This paper reports the results of an empirical study that use a mixed-method approach to investigate the adoption of the exception handing mechanism in C++. Firstly, we carried out a static analysis investigation to understand how developers employ the exception handling construct of C++, considering 65 open-source systems (which comprise 34 million lines of C++ code overall). Then, to better understand the findings from the static analysis phase, we conducted a survey involving 145 C++ developers who have contributed to the subject systems. Some of the findings consistently detected during this mixed-method study reveal that, for several projects, the use of exception handling constructs is scarce and developers favor the use of other strategies to deal with exceptional conditions. In addition, the survey respondents consider that incompatibility with existing C code and libraries, extra performance costs (in terms of response time and size of the compiled code), and lack of expertise to design an exception handling strategy are among the reasons for avoiding the use of exception handling constructs.

The exception handling policy of a system comprises the set of design rules that specify its exception handling behavior (how exceptions should be handled and thrown).Such policy is usually undocumented and implicitly defined by the system architect.For this reason, developers may think that by just including catch-blocks in the code they can deal with exception conditions.This lack of information may turn the exception handling into a generalized "goto" mechanism making the program more complex and less reliable.This work proposes a domain-specific language called ECL (Exception Contract Language) to specify the exception handling policy and a runtime monitoring tool which dynamically checks this policy.The monitoring tool is implemented in the form of an aspect library, which can be added to any Java system without the need to change the application source code.We applied this approach to a large-scale web-based system and to a set of versions of the well-known JUnit framework.The results indicate that this approach can be used to express and to automatically check the exception handling policy of a system, and consequently support the development of more robust Java systems.

The exception handling policy of a system comprises the set of design rules that specify its exception handling behavior (how exceptions should be handled and thrown). Such policy is usually undocumented and implicitly defined by the system architect. For this reason, developers often consider that by just including catch-blocks in the code they are dealing with exceptional conditions. This lack of information may turn the exception handling into a generalized “goto” mechanism making the program more complex and less reliable. This work presents a domain-specific language called ECL (Exception Contract Language) to specify the exception handling policy and a runtime monitoring tool which dynamically checks this policy. The monitoring tool is implemented in the form of an aspect library, which can be added to any Java system without the need to change the application source code. We applied this approach to two large-scale web-based systems and to a set of versions of the well-known JUnit framework. The results indicate that this approach can be used to express and to automatically check the exception handling policy of a system, and consequently support the development of more robust Java systems.

There is a growing interest of the Computer Science education community for including testing concepts on introductory programming courses. Aiming at contributing to this issue, we introduce POPT, a Problem-Oriented Programming and Testing approach for Introductory Programming Courses. POPT main goal is to improve the traditional method of teaching introductory programming that concentrates mainly on implementation and neglects testing. According to POPT, students' skills must be developed by dealing with ill-defined problems, from which students are stimulated to develop test cases in a table-like manner in order to enlighten the problems' requirements and also to improve the quality of generated code. This paper presents POPT and a case study performed in an Introductory Programming course of a Computer Science program at the Federal University of Rio Grande do Norte, Brazil. The study results have shown that, when compared to a Blind Testing approach, POPT stimulates the implementation of programs of better external quality - the first program version submitted by POPT students passed in twice the number of test cases (professor-defined ones) when compared to non-POPT students. Moreover, POPT students submitted fewer program versions and spent more time to submit the first version to the automatic evaluation system, which lead us to think that POPT students are stimulated to think better about the solution they are implementing.

Although it is assumed that the implementation of design by contract is better modularized by means of aspect-oriented (AO) programming, there is no empirical evidence on the effectiveness of AO for modularizing non-trivial design by contract code in realistic development scenarios. This paper reports a quantitative and qualitative case study that evolves a real-life application to assess various facets of the adequacy of aspects for modularizing the design by contract concern. Our evaluation focused upon a number of system changes that are typically performed during software maintenance tasks. The study was driven by an analysis of fundamental modularity attributes, such as separation of concerns, coupling, conciseness, and change propagation. We have found that AO techniques improved separation of concerns and the design stability between the design by contract code and base application code throughout the development scenarios. However, contradicting the general intuition, the AO versions of the system did not present significant gains regarding four classical size metrics we employed.

Software product line engineering is about producing a set of related products that share more commonalities than variabilities. This approach promotes benefits such as cost reduction, product quality, productivity and time to market, but it brings new challenges that must be considered during the evolution of the software product line. In this context, recent research has explored and proposed automated approaches based on code analysis and traceability techniques for change impact analysis. This paper presents Squid, an extensible infrastructure for analyzing software product line implementations. The approach uses information from variability modeling, variability mapping to code assets, and dependency relationships between code assets to perform analysis of SPL implementations. A Squid instantiation example is presented to illustrate the usage of the tool in practical scenarios.

Design by Contract (DbC) is a technique for developing and improving functional software correctness through definition of "contracts" between client classes and their suppliers. Such contracts are enforced during runtime and if any of them is violated a runtime error should occur. Runtime assertions checkers (RACs) are a well-known technique that enforces such contracts. Although they are largely used to implement the DbC technique in contemporary languages, like Java, studies have shown that characteristics of contemporary exception handling mechanisms can discard contract violations detected by RACs. As a result, a contract violation may not be reflected in a runtime error, breaking the supporting hypothesis of DbC. This paper presents an error recovery technique for RACs that tackles such limitations. This technique relies on aspect-oriented programming in order to extend the functionalities of existing RACs stopping contract violations from being discarded. We applied the recovery technique on top of five Java-based contemporary RACs (i.e., JML/jml, JML/ajml, JContractor, CEAP, and Jose). Preliminary results have shown that the proposed technique could actually prevent the contract violations from being discarded regardless of the characteristics of the exception handling code of the target application.

Automated tests have been widely used as a supporting mechanism during software development and maintenance activities. It improves the confidence on software releases as it seeks to uncover regression bugs, and serves as a live documentation which is very useful when evolving systems. Concerning multi agent systems (MASs), some characteristics such as agent autonomy and asynchronous message-based interaction bring a degree of non-determinism which presents new testing challenges. This paper proposes JAT, a framework for building and running MASs test scenarios, which relies on the use of aspect-oriented techniques to monitor the autonomous agents during tests and control the test input of asynchronous test cases. The tool has been developed on top of JADE, a widely used agent platform implemented in Java. We have used JAT on testing 3 different MASs. Our experience shows that JAT can be used to build test scenarios which can achieve high fault-detection effectiveness.

One of the main promises of aspect-oriented programming (AOP) is to promote improved modularization of crosscutting concerns, thereby enhancing the software stability in the presence of changes. This paper presents a quantitative study that assesses the positive and negative effects of AOP on typical maintenance activities of a Web information system. The study consists of a systematic comparison between the object-oriented and the aspect-oriented versions of the same application in order to assess to what extent each solution provides maintainable software decompositions. Our analysis was driven by fundamental modularity attributes, such as coupling, cohesion, conciseness, and separation of concerns. We have found that the aspect-oriented design has exhibited superior stability and reusability through the changes, as it has resulted in fewer lines of code, improved separation of concerns, weaker coupling, and lower intra-component complexity