Background. Test smells can pose difficulties during testing activities, such as poor maintainability, non-deterministic behavior, and incomplete verification. Existing research has extensively addressed test smells in automated software tests, but little attention has been paid to smells in natural language tests. While some research has attempted to catalog such test smells, there is a lack of investigation into their impact on the effectiveness of test cases. Aims. In this paper, we conduct a controlled experiment with 30 participants from academia and industry to examine the impact of test smells in manual test descriptions. Method. Specifically, we analyze whether the presence of two test smells, Ambiguous Test and Eager Action, result in (1) increased test execution time, (2) a higher number of steps needed to complete the tests, and (3) high divergency on the perceived success of the tests outcomes. Results. Our findings reveal that an Ambiguous Test can increase execution time by up to five times and screen flow by up to seven times. In addition, if the Eager Actions are dependent on one another, there is no increase in execution time and screen flow. Conclusions. It highlights the need for better design of manual test descriptions to improve clarity, consistency, and performance execution.

Abstract Test smells are considered bad practices that can reduce the test code quality, thus harming software testing goals and maintenance activities. Prior studies have investigated the diffusion of test smells and their impact on test code maintainability. However, we cannot directly compare the outcomes of the studies as most of them use customized datasets. In response, we introduced the TSSM (Test Smells and Structural Metrics) dataset, containing test smells detected using the JNose Test tool and structural metrics (test code and production code) calculated with the CK metrics tool of 13,703 open‐source Java systems from GitHub. In addition, we perform an empirical study to investigate the relationship between test smells and structural metrics of test code and the relationship between test smells on a large‐scale dataset. We split the projects into three clusters to analyze the distribution of test smells, the co‐occurrences among test smells, and the correlation of test smells and structural metrics of test code. The ratio of smelly test classes with a specific test smell is similar among the clusters, but we could observe a significant difference in the number of test smells among them. The test smells Sleepy Test, Mystery Guest, and Resource Optimism rarely occur in the three clusters, and the last two are strongly correlated, indicating that those test smells are more severe than others. Our results point out that most test smells have a moderate correlation with high complexity, large size, and coupling of the test code, indicating that they can also negatively affect its quality. To support further studies, we made our dataset publicly available.

Refactorings are transformations to improve the code design without changing overall functionality and observable behavior. During the refactoring process of smelly test code, practitioners may struggle to identify refactoring candidates and define and apply corrective strategies. This paper reports on an empirical study aimed at understanding how test smells and test refactorings are discussed on the Stack Exchange network. Developers commonly count on Stack Exchange to pick the brains of the wise, i.e., to ‘look up’ how others are completing similar tasks. Therefore, in light of data from the Stack Exchange discussion topics, we could examine how developers understand and perceive test smells, the corrective actions they take to handle them, and the challenges they face when refactoring test code aiming to fix test smells. We observed that developers are interested in others’ perceptions and hands-on experience handling test code issues. Besides, there is a clear indication that developers often ask whether test smells or anti-patterns are either good or bad testing practices than code-based refactoring recommendations.

Refactoring is a practice commonly used by developers to restructure the source code without changing its external behavior. Over the last decades, the software engineering research community has been making use of mining software repository techniques to investigate refactoring under multiple perspectives, identifying properties and impact of this practice on source code quality, other than using refactoring data coming from software repositories to build automated recommendation systems. While the current state of the art proposes various automated tools to mine refactoring data, there is still a lack of instruments that may help researchers when mining test-specific refactoring data. The availability of those instruments may enable additional, specialized techniques to support developers while refactoring test code. In this paper, we introduce an approach that extends REFACTORINGMINER-a well-established refactoring mining tool having high precision and recall scores- and is able to detect seven test-specific refactoring operations. We perform mixed-method research to assess capabilities and usefulness of the approach. First, we compare the test-specific refactoring data extracted by the approach against an oracle of 375 test-specific refactorings. Second, we engage with 15 software engineering researchers and apply a technology acceptance model to investigate how they would benefit from our approach. The key results of the study show that our approach reaches 100% and 92.5% of precision and recall scores, respectively. In addition, the approach is considered useful and suitable for various research tasks, including the definition of novel learning models able to recommend test-specific refactoring actions.

The teaching-learning process may require specific pedagogical approaches to establish a relationship with industry practices. Recently, some studies investigated the educators’ perspectives and the undergraduate courses curriculum to identify potential weaknesses and solutions for the software testing teaching process. However, it is still unclear how the practitioners evaluate the acquisition of knowledge about software testing in undergraduate courses. This study carried out an expert survey with 68 newly graduated practitioners to determine what the industry expects from them and what they learned in academia. The yielded results indicated that those practitioners learned at a similar rate as others with a long industry experience. Also, they studied less than half of the 35 software testing topics collected in the survey and took industry-backed extracurricular courses to complement their learning. Additionally, our findings point out a set of implications for future research, as the respondents’ learning difficulties (e.g., lack of learning sources) and the gap between academic education and industry expectations (e.g., certifications).

Software product line (SPL) is one of the most recent and effective reuse approaches. SPL derives several products from the core artefacts. SPL engineering includes two processes: domain engineering, which identifies the common and variable features to develop the core artefacts, and application engineering, which reuses the core artefacts to derive products. Once the artefacts are reused across multiple products, quality assessment is necessary to prevent inconsistencies from spreading across all SPL products. There are several frameworks and standards, as ISO/IEC 25010:2011, to evaluate quality characteristics. In this study, the authors provide an overview of the SPL quality assessment. Therefore, they perform a systematic mapping to compile and synthesise data regarding the quality characteristics assessed in studies from 2000 to 2019. The results include the identification of 346 metrics applied in 16 software properties to evaluate three quality characteristics of the ISO/IEC 25010:2011. Additionally, they find the domain engineering evaluation frequently occurs regarding the maintainability characteristic. Moreover, they provide analyses of the: (i) metrics used by programming paradigm, (ii) metrics used by software properties, (iii) software properties evaluated for each quality characteristic, (iv) tools used to extract metrics, (v) systems used as benchmarks, and (vi) datasets used for extracting metrics.