Confounders (or confounding variables) pose significant challenges to detecting reliable causal relationships in observational studies. When data are collected from naturally occurring phenomena—e.g., mining software repositories (MSR)—researchers cannot rely on randomization to control for confounders, leading to biased causal inferences. Alternative approaches are required to mitigate confounding bias when exploring causal inferences. This paper explains and exemplifies the use of matching in MSR.

Abstract [Context] Microservices enable the decomposition of applications into small, independent, and connected services. The independence between services could positively affect a project’s velocity, which is considered an important maintenance metric measuring the time taken to implement features and fix bugs. However, no studies have investigated the causal relationship between microservices and velocity. [Objective and Method] The goal of this study is to investigate the effect of microservices on velocity which is a common maintenance metric. The study compares projects on GitHub developed with microservices style from the beginning and similar projects using monolithic architectures. The study was conducted as a retrospective cohort study, which is a study type used to assess causality. [Results] The results did not find statistically significant differences in mean velocities in microservice-based and monolithic projects. Furthermore, the statistical adjustment performed to quantify the statistical impact of the use of microservices on velocity considering additional confounders did not find statistically significant impact from these. [Conclusions] The results did not indicate a difference between microservices-based projects and monolithic projects in terms of velocity. In addition, this study will contribute to the body of knowledge of empirical methods and be among the first works to adopt the methodology of the cohort study.

Background: Code Technical Debt (Code TD) prediction has gained significant attention in recent software engineering research. However, no standardized approach to Code TD prediction fully captures the factors influencing its evolution. Objective: Our study aims to assess the impact of time-dependent models and seasonal effects on Code TD prediction. It evaluates such models against widely used Machine Learning models also considering the influence of seasonality on prediction performance. Methods: We trained 11 prediction models with 31 Java open-source projects. To assess their performance, we predicted future observations of the SQALE index. To evaluate the practical usability of our TD forecasting model and their impact on practitioners, we surveyed 23 software engineering professionals. Results: Our study confirms the benefits of time-dependent techniques, with the ARIMAX model outperforming the others. Seasonal effects improved predictive performance, though the impact remained modest. ARIMAX/SARIMAX models demonstrated to provide well-balanced long-term forecasts. The survey highlighted strong industry interest in short- to medium-term TD forecasts. Conclusions: Our findings support using techniques that capture time dependence in historical software metric data, particularly for Code TD. Effectively addressing this evidence requires adopting methods that account for temporal patterns.

Mining Software Repositories studies have become increasingly popular over the years. However, a notable limitation is that they report correlational relationships rather than establishing causation. In contrast, certain disciplines (e.g. epidemiology) have developed specific methods to address this limitation. The goal of this tutorial is to introduce participants to one such method: cohort studies. By the end of the tutorial, participants will be familiar with the steps and techniques involved in designing and analyzing cohort studies.

Predicting Technical Debt has become a popular research niche in recent software engineering literature. However, there is no consistent approach yet that succeeds in entirely capturing the nature of this type of data. We applied each technique on a dataset consisting of the commit data of a total of 28 Java projects. We predicted the future values of the SQALE index to evaluate their predictive performance. Using these techniques we confirmed the predictive power of each of them with the same commit data. We aim to investigate further the time-dependent nature of other types of commit data to validate the existing prediction techniques.

Developers use Static Analysis Tools (SATs) to control for potential quality issues in source code, including defects and technical debt. Tool vendors have devised quite a number of tools, which makes it harder for practitioners to select the most suitable one for their needs. To better support developers, researchers have been conducting several studies on SATs to favor the understanding of their actual capabilities. Despite the work done so far, there is still a lack of knowledge regarding (1) what is their agreement, and (2) what is the precision of their recommendations. We aim at bridging this gap by proposing a large-scale comparison of six popular SATs for Java projects: Better Code Hub, CheckStyle, Coverity Scan, FindBugs, PMD, and SonarQube. We analyze 47 Java projects applying 6 SATs. To assess their agreement, we compared them by manually analyzing – at line – and class-level — whether they identify the same issues. Finally, we evaluate the precision of the tools against a manually-defined ground truth. The key results show little to no agreement among the tools and a low degree of precision. Our study provides the first overview on the agreement among different tools as well as an extensive analysis of their precision that can be used by researchers, practitioners, and tool vendors to map the current capabilities of the tools and envision possible improvements.

Background. Several recent software engineering studies use data mined from the version control systems adopted by the different software projects. However, inspecting the data and statistical methods used in those studies reveals several problems with the current approach, mainly related to the dependent nature of the data. Objective. We analyzed time-dependent data in software engineering at commit level, and propose an alternative approach based on time series analysis. Method. We identified statistical tests designed for time series analysis and propose a technique to model time dependent data, similarly to what is done in finance and weather forecasting. We applied our approach to a small set of projects of different sizes, investigating the behaviour of the SQALE Index, in order to highlight the time and interdependency of the different commits. Results. Using these techniques, we analysed and model the data, showing that it is possible to investigate this type of commit data using methods from time series analysis. Conclusion. Based on the promising results, we plan to validate the robustness of the approach by replicating previous works.

Background: Technical Debt is a consolidated notion in software engineering research and practice. However, the estimation of its impact (interest of the debt) is still imprecise and requires heavy empirical and experimental inquiry. Objective: We aim at developing a data-driven approach to calculate the interest of Technical Debt in terms of delays in resolving affected tasks.Method: We conducted a case study to estimate the Technical Debt interest by analyzing its association with the lead time variation of resolving related Jira issues.Results: Data-driven approaches could significantly change the Technical Debt estimation and improve the removing Technical Debt prioritization. Our case study shows that the presence of Code Technical Debt did not affect the lead time for resolving the issues.Conclusion: Future works include the further refinement of this approach and its application to a larger data-set and on different type of issues.

Pull requests are a common practice for making contributions and reviewing them in both open-source and industrial contexts. Our goal is to understand whether quality flaws such as code smells, anti-patterns, security vulnerabilities, and coding style violations in a pull request’s code affect the chance of its acceptance when reviewed by a maintainer of the project. We conducted a case study among 28 Java open-source projects, analyzing the presence of 4.7 M code quality flaws in 36 K pull requests. We analyzed further correlations by applying logistic regression and six machine learning techniques. Moreover, we manually validated 10% of the pull requests to get further qualitative insights on the importance of quality issues in cases of acceptance and rejection. Unexpectedly, quality flaws measured by PMD turned out not to affect the acceptance of a pull request at all. As suggested by other works, other factors such as the reputation of the maintainer and the importance of the delivered feature might be more important than other qualities in terms of pull request acceptance. . Researchers have already investigated the influence of the developers’ reputation and the pull request acceptance. This is the first work investigating code style violations and specifically PMD rules. We recommend that researchers further investigate this topic to understand if different measures or different tools could provide some useful measures.

Background. Most Mining Software Repositories (MSR) studies cannot obtain causal relations because they are not controlled experiments. The use of cohort studies as defined in epidemiology could help to overcome this shortcoming.

Background. The need to release our products under tough time constraints has required us to take shortcuts during the implementation of our products and to postpone the correct implementation, thereby accumulating Technical Debt. Objective. In this work, we report the experience of a Finnish SME in managing Technical Debt (TD), investigating the most common types of TD they faced in the past, their causes, and their effects. Method. We set up a focus group in the case-company, involving different roles. Results. The results showed that the most significant TD in the company stems from disagreements with the supplier and lack of test automation. Specification and test TD are the most significant types of TD. Budget and time constraints were identified as the most important root causes of TD. Conclusion. TD occurs when time or budget is limited or the amount of work are not understood properly. However, not all postponed activities generated ”debt”. Sometimes the accumulation of TD helped meet deadlines without a major impact, while in other cases the cost for repaying the TD was much higher than the benefits. From this study, we learned that learning, careful estimations, and continuous improvement could be good strategies to mitigate TD These strategies include iterative validation with customers, efficient communication with stakeholders, meta-cognition in estimations, and value orientation in budgeting and scheduling.

[Context] The popularity of tools for software quality analysis has increased over the years, with special attention to tools that calculate technical debt based on violations of a set of rules. SonarQube is one of the most used tools and provides an estimation of the time needed to remediate technical debt. However, practitioners are still skeptical about the accuracy of its remediation time estimation. [Objective] In this paper, we analyze the accuracy of SonarQube remediation time on a set of 15 open source Java projects. [Method] We designed and conducted a case study where we asked 65 novice developers to remove rule violations and reduce the technical debt of 15 projects. [Results] The results point out that SonarQube remediation time, compared to the actual time for reducing technical debt, is generally overestimated, and that the most accurate estimation relates to code smells, while the least accurate concerns bugs. [Conclusions] Practitioners and researchers could benefit from the results of this work to understand up to which extent technical debt is overestimated and have a more accurate estimation of the remediation time.?

Background. Companies commonly invest major effort into removing, respectively not introducing, technical debt issues detected by static analysis tools such as SonarQube, Cast, or Coverity. These tools classify technical debt issues into categories according to severity, and developers commonly pay attention to not introducing issues with a high level of severity that could generate bugs or make software maintenance more difficult. Objective. In this work, we aim to understand the diffuseness of Technical Debt (TD) issues and the speed with which developers remove them from the code if they introduced such an issue. The goal is to understand which type of TD is more diffused and how much attention is paid by the developers, as well as to investigate whether TD issues with a higher level of severity are resolved faster than those with a lower level of severity. We conducted a case study across 78K commits of 33 Java projects from the Apache Software Foundation Ecosystem to investigate the distribution of 1.4M TD items. Results. TD items introduced into the code are mostly related to code smells (issues that can increase the maintenance effort). Moreover, developers commonly remove the most severe issues faster than less severe ones. However, the time needed to resolve issues increases when the level of severity increases (minor issues are removed faster that blocker ones). Conclusion. One possible answer to the unexpected issue of resolution time might be that severity is not correctly defined by the tools. Another possible answer is that the rules at an intermediate severity level could be the ones that technically require more time to be removed. The classification of TD items, including their severity and type, require thorough investigation from a research point of view.