Abstract Mocking frameworks provide convenient APIs, which create mock objects, manipulate their behavior, and verify their execution, for the purpose of isolating test dependencies in unit testing. This study contributes an in-depth empirical study of whether and how mocking frameworks are used in Apache projects. The key findings and insights of this study include: First, mocking frameworks are widely used in 66% of Apache Java projects, with Mockito, EasyMock, and PowerMock being the top three most popular frameworks. Larger-scale and more recent projects tend to observe a stronger need to use mocking frameworks. This underscores the importance of mocking in practice and related future research. Second, mocking is overall practiced quite selectively in software projects—not all test files use mocking, nor all dependencies of a test target are mocked. It calls for more future research to gain a more systematic understanding of when and what to mock to provide formal guidance to practitioners. On top of this, the intensity of mocking in different projects shows different trends in the projects’ evolution history—implying the compound effects of various factors, such as the pace of a project’s growth, the available resources, time pressure, and priority, etc. This points to an important future research direction in facilitating best mocking practices in software evolution. Furthermore, we revealed the most frequently used APIs in the three most popular frameworks, organized based on the function types. The top five APIs in each functional type of the three mocking frameworks usually take the majority (78% to 100%) of usage in Apache projects. This indicates that developers can focus on these APIs to quickly learn the common usage of these mocking frameworks. We further investigated informal methods of mocking, which do not rely on any mocking framework. These informal mocking methods point to potential sub-optimal mocking practices that could be improved, as well as limitations of existing mocking frameworks. Finally, we conducted a developer survey to collect additional insights regarding the above analysis based on their experience, which complements our analysis based on repository mining. Overall, this study offers practitioners profound empirical knowledge of how mocking frameworks are used in practice and sheds light on future research directions to enhancing mocking in practice.

Unit testing focuses on verifying the functions of individual units of a software system. It is challenging due to the high inter dependencies among software units. Developers address this by mocking—replacing the dependency by a “fake” object. Despite the existence of powerful, dedicated mocking frameworks, developers often turn to a “hand-rolled” approach—inheritance. That is, they create a subclass of the dependent class and mock its behavior through method overriding. However, this requires tedious implementation and compromises the design quality of unit tests. This work contributes a fully automated refactoring framework to identify and replace the usage of inheritance by using Mockito—a well received mocking framework. Our approach is built upon the empirical experience fromfiveopen source projects that use inheritance for mocking. We evaluate our approach onnineother projects. Results show that our framework is efficient, generally applicable to new datasets, mostly preserves test case behaviors in detecting defects (in the form of mutants), and decouples test code from production code. The qualitative evaluation by experienced developers suggests that the auto-refactoring solutions generated by our framework improve the quality of the unit test cases in various aspects, such as making test conditions more explicit, as well as improved cohesion, readability, understandability, and maintainability with test cases. Finally, we submit 23 pull requests containing our refactoring solutions to the open source projects. It turns our that, 9 requests are accepted/merged, 6 requests are rejected, the remaining requests are pending (5 requests), with unexpected exceptions (2 requests), or undecided (1 request). In particular, among the 21 open source developers that are involved in the reviewing process, 81% give positive votes. This indicates that our refactoring solutions are quite well received by the open source projects and developers.

TheAAApattern (i.e.,Arrange-Act-Assert) is a common and natural layout to create a test case. Following this pattern in test cases may benefit comprehension, debugging, and maintenance. TheAAAstructure of real-life test cases, however, may not be clear due to their high complexity. Manually labelingAAAstatements in test cases is tedious. Thus, we envision that an automated approach for labelingAAAstatements in existing test cases could benefit new developers and projects that practice collective code ownership and test-driven development. This paper contributes an automatic approach based on machine learning models. The “secret sauce” of this approach is a set of three learning features that are based on the semantic, syntax, and context information in test cases, derived from the manual tagging process. Thus, our approach mimics how developers may manually tag theAAApattern of a test case. We assess the precision, recall, and F-1 score of our approach based on 449 test cases, containing about 16,612 statements, across 4 Apache open source projects. To achieve the best performance in our approach, we explore the usage of six machine learning models; the contribution of the SMOTE data balancing technique; the comparison of the three learning features; and the comparison of five different methods for calculating the semantic feature. The results show our approach is able to identifyArrangement,Action, andAssertionstatements with a precision upwards of 92%, and recall up to 74%. We also summarize some experience based on our experiments—regarding the choice of machine learning models, data balancing algorithm, and feature engineering methods—which could potentially provide some reference to related future research.

The AAA pattern, i.e. the Arrangement, Action, and Assertion, is a common and nature layout to create a test case. Following this pattern in test cases may benefit comprehension, debugging, and maintenance. The AAA structure of real-life test cases may not be explicit due to its high complexity. Manually labeling AAA statements in test cases is tedious. Thus, an automated approach for labeling AAA statements in existing test cases could benefit new developers and projects that practice collective code ownership and test driven development.

Unit testing focuses on verifying the functions of individual units of a software system. It is challenging due to the high inter-dependencies among software units. Developers address this by mocking-replacing the dependency by a "faked" object. Despite the existence of powerful, dedicated mocking frameworks, developers often turn to a "hand-rolled" approach-inheritance. That is, they create a subclass of the dependent class and mock its behavior through method overriding. However, this requires tedious implementation and compromises the design quality of unit tests. This work contributes a fully automated refactoring framework to identify and replace the usage of inheritance by using Mockito-a well received mocking framework. Our approach is built upon the empirical experience from five open source projects that use inheritance for mocking. We evaluate our approach on four other projects. Results show that our framework is efficient, generally applicable to new datasets, mostly preserves test case behaviors in detecting defects (in the form of mutants), and decouples test code from production code. The qualitative evaluation by experienced developers suggests that the auto-refactoring solutions generated by our framework improve the quality of the unit test cases in various aspects, such as making test conditions more explicit, as well as improved cohesion, readability, understandability, and maintainability with test cases.

The design structure of a system continuously evolves as the consequence of fast-paced code revisions. Agile techniques, such as continuous testing, ensures the function goals of a system with every code revision. However, there lacks an efficient approach that can continuously model the design difference resulting from every single code revision to facilitate comprehension and ensure the design quality. This paper contributes a novel design modeling approach, called Design Differencing (DESIGNDIFF), that models and visualizes the highlevel design differences resulting from every code revision. This paper defines a complete and general set of 17 design change operators to capture the design difference from any code revision. We evaluated the potential of DESIGNDIFF in three aspects. First, a user study of 10 developers indicated that DESIGNDIFF can help practitioners to faster and better understand high-level design differences from real-life software commits. Second, DESIGNDIFF analyzed 14,832 real-life commits in five real-life projects: finding 4,189 commits altered the software design, 855 commits introduced and 337 commits eliminated design flaws. The latency between the flaw introduction and elimination is on average 2 months to 2 years! With an affordable performance overhead, DESIGNDIFF has great potential to benefit practitioners in more applications.

Performance issues widely exist in modern software systems. Existing performance optimization approaches, such as dynamic profiling, usually fail to consider the impacts of architectural connections among methods on performance issues. This paper contributes an architectural approach, Butterfly Space modeling, to investigate performance issues. Each Butterfly Space is composed of 1) a seed method; 2) methods in the "upper wing" that call the seed directly or transitively; and 3) methods in the "lower wing" that are called by the seed, directly or transitively. The rationale is that the performance of the seed method impacts and is impacted by all the other methods in the space because of the call relationship. As such, developers can more efficiently investigate groups of connected performance improvement opportunities in Butterfly Spaces. We studied three real-world open source Java projects to evaluate such potential. Our findings are three-fold: 1) If the seed method of a Butterfly Space contains performance problems, up to 60% of the methods in the space also contain performance problems; 2) Butterfly Spaces can potentially help to non-trivially increase the precision/recall and reduce the costs in identifying performance improvement opportunities, compared to dynamic profiling; and 3) Visualizing dynamic profiling metrics with Butterfly Spaces simultaneously help to reveal two typical patterns, namely Expensive Callee and Inefficient Caller, that are responsible for performance problems and provide insights on where to improve next. We believe that Butterfly Space modeling has great potential for investigating performance issues.

Performance issues compromise the response time and resource consumption of a software system. Modern software systems use issue tracking systems to manage all kinds of issue reports, including performance issues. The problem is that performance issues are often not explicitly tagged. The tagging mechanism, if exists, is completely voluntary, depending on the project’s convention and on submitters’ discipline. For example, the performance tag rate in Apache’s Jira system is below 1%. This paper contributes a hybrid classification approach that combines linguistic patterns and machine/deep learning techniques to automatically detect performance issue reports. We manually analyzed 980 real-life performance issue reports and derived 80 project-agnostic linguistic patterns that recur in the reports. Our approach uses these linguistic patterns to construct the sentence-level and issue-level learning features for training effective machine/deep learning classifiers. We test our approach on two separate datasets, each consisting of 980 unclassified issue reports, and compare the results with 31 baseline methods. Our approach can reach up to 83% precision and up to 59% recall. The only comparable baseline method is BERT, which is still 25% lower in the F1-score.

Empirical experience regarding how real-life performance issues are caused and resolved can provide valuable insights for practitioners to effectively and efficiently prevent, detect, and fix performance issues. Prior work shows that most performance issues have their roots in poor architectural decisions. This paper contributes a large scale empirical study of 192 real-life performance issues, with an emphasis on software design. First, this paper contributes a holistic view of eight common root causes and typical resolutions that recur in different projects, and surveyed existing literature, in particular, tools, that can detect and fix each type of performance issue. Second, this study is first-of-its-kind to investigate performance issues from a design perspective. In the 192 issues, 33% required design-level optimization, i.e. simultaneously revising a group of related source files for resolving the issues. We reveal four design-level optimization patterns, which have shown different prevalence in resolving different root causes. Finally, this study investigated the Return on Investment for addressing performance issues, to help practitioners choose between localized or design-level optimization resolutions, and to prioritize issues due to different root causes.

Performance problems widely exist in modern software systems. Existing performance optimization techniques, including profiling-based and pattern-based techniques, usually fail to consider the architectural impacts among methods that easily slow down the overall system performance. This paper contributes a new approach, named Speedoo, to identify groups of methods that should be treated together and deserve high priorities for performance optimization. The uniqueness of Speedoo is to measure and rank the performance optimization opportunities of a method based on 1) the architectural impact and 2) the optimization potential. For each highly ranked method, we locate a respective Optimization Space based on 5 performance patterns generalized from empirical observations. The top ranked optimization spaces are suggested to developers as potential optimization opportunities. Our evaluation on three real-life projects has demonstrated that 18.52% to 42.86% of methods in the top ranked optimization spaces indeed undertook performance optimization in the projects. This outperforms one of the state-of-the-art profiling tools YourKit by 2 to 3 times. An important implication of this study is that developers should treat methods in an optimization space together as a group rather than as individuals in performance optimization. The proposed approach can provide guidelines and reduce developers' manual effort.