The AAA (Arrange, Act, Assert) pattern provides a unified structure for unit test cases, potentially benefiting comprehension and maintenance. However, its adoption and implementation in practice remain insufficiently understood. This study investigates the prevalence of AAA pattern usage, identifies recurring deviations and design issues within AAA structures, and assesses developers’ receptiveness to AAA-based improvements. We conducted an empirical study on 735 real-life unit test cases randomly selected from seven open-source projects. We manually analyzed these test cases, identified AAA-related issues, and proposed fixes to developers. Our analysis found that 77% of test cases follow the AAA structure. We identified three recurring patterns deviating from AAA and four design issues within A blocks. Comparison with classic test smells revealed unique insights provided by AAA analysis. Of 27 improvement proposals sent to developers, 78% received positive feedback. These findings show that the AAA pattern is widely adopted in practice, but deviations from and design issues within AAA patterns are common. Our analysis provides a novel perspective on test case quality, complementing traditional test smell analysis. The high acceptance rate of our improvement proposals suggests that developers value AAA-based enhancements. These findings can guide the development of tools for improving AAA practice in unit tests.

Abstract Mocking frameworks provide convenient APIs, which create mock objects, manipulate their behavior, and verify their execution, for the purpose of isolating test dependencies in unit testing. This study contributes an in-depth empirical study of whether and how mocking frameworks are used in Apache projects. The key findings and insights of this study include: First, mocking frameworks are widely used in 66% of Apache Java projects, with Mockito, EasyMock, and PowerMock being the top three most popular frameworks. Larger-scale and more recent projects tend to observe a stronger need to use mocking frameworks. This underscores the importance of mocking in practice and related future research. Second, mocking is overall practiced quite selectively in software projects—not all test files use mocking, nor all dependencies of a test target are mocked. It calls for more future research to gain a more systematic understanding of when and what to mock to provide formal guidance to practitioners. On top of this, the intensity of mocking in different projects shows different trends in the projects’ evolution history—implying the compound effects of various factors, such as the pace of a project’s growth, the available resources, time pressure, and priority, etc. This points to an important future research direction in facilitating best mocking practices in software evolution. Furthermore, we revealed the most frequently used APIs in the three most popular frameworks, organized based on the function types. The top five APIs in each functional type of the three mocking frameworks usually take the majority (78% to 100%) of usage in Apache projects. This indicates that developers can focus on these APIs to quickly learn the common usage of these mocking frameworks. We further investigated informal methods of mocking, which do not rely on any mocking framework. These informal mocking methods point to potential sub-optimal mocking practices that could be improved, as well as limitations of existing mocking frameworks. Finally, we conducted a developer survey to collect additional insights regarding the above analysis based on their experience, which complements our analysis based on repository mining. Overall, this study offers practitioners profound empirical knowledge of how mocking frameworks are used in practice and sheds light on future research directions to enhancing mocking in practice.

TheAAApattern (i.e.,Arrange-Act-Assert) is a common and natural layout to create a test case. Following this pattern in test cases may benefit comprehension, debugging, and maintenance. TheAAAstructure of real-life test cases, however, may not be clear due to their high complexity. Manually labelingAAAstatements in test cases is tedious. Thus, we envision that an automated approach for labelingAAAstatements in existing test cases could benefit new developers and projects that practice collective code ownership and test-driven development. This paper contributes an automatic approach based on machine learning models. The “secret sauce” of this approach is a set of three learning features that are based on the semantic, syntax, and context information in test cases, derived from the manual tagging process. Thus, our approach mimics how developers may manually tag theAAApattern of a test case. We assess the precision, recall, and F-1 score of our approach based on 449 test cases, containing about 16,612 statements, across 4 Apache open source projects. To achieve the best performance in our approach, we explore the usage of six machine learning models; the contribution of the SMOTE data balancing technique; the comparison of the three learning features; and the comparison of five different methods for calculating the semantic feature. The results show our approach is able to identifyArrangement,Action, andAssertionstatements with a precision upwards of 92%, and recall up to 74%. We also summarize some experience based on our experiments—regarding the choice of machine learning models, data balancing algorithm, and feature engineering methods—which could potentially provide some reference to related future research.

The AAA pattern, i.e. the Arrangement, Action, and Assertion, is a common and nature layout to create a test case. Following this pattern in test cases may benefit comprehension, debugging, and maintenance. The AAA structure of real-life test cases may not be explicit due to its high complexity. Manually labeling AAA statements in test cases is tedious. Thus, an automated approach for labeling AAA statements in existing test cases could benefit new developers and projects that practice collective code ownership and test driven development.