Code readability is fundamental to software quality and maintainability. Poor readability extends development time, increases bug-inducing risks, and contributes to technical debt. With the rapid advancement of Large Language Models, AI agent-based approaches have emerged as a promising paradigm for automated refactoring, capable of decomposing complex tasks through autonomous planning and execution. While prior studies have examined refactoring by AI agents, these analyses cover all forms of refactoring, including performance optimization and structural improvement. As a result, the extent to which AI agent-based refactoring specifically improves code readability remains unclear.

Policy as Code (PaC) is an emerging DevOps practice that enables teams to specify organisational and technical policies, such as regulatory compliance, security requirements, and resource limits, through machine-enforceable declarative code. As PaC gains prominence, practitioners face difficulties in adopting PaC while there remains a limited empirical understanding of how these policies are introduced, what types can be expressed, and how they are maintained in practice.

Fuzzing has become a popular technique for automatically detecting vulnerabilities and bugs by generating unexpected inputs. In recent years, the fuzzing process has been integrated into continuous integration workflows (i.e., continuous fuzzing), enabling short and frequent testing cycles. Despite its widespread adoption, prior research has not examined whether the effectiveness of continuous fuzzing varies across programming languages.

Agentic Coding, powered by autonomous agents such as GitHub Copilot and Cursor, enables developers to generate code, tests, and pull requests from natural language instructions alone. While this accelerates implementation, it produces larger volumes of code per pull request, shifting the burden from implementers to reviewers. In practice, a notable portion of AI-generated code is eventually deleted during review, yet reviewers must still examine such code before deciding to remove it. No prior work has explored methods to help reviewers efficiently identify code that will be removed.

Agent-based coding tools have transformed software development practices. Unlike prompt-based approaches that require developers to manually integrate generated code, these agent-based tools autonomously interact with repositories to create, modify, and execute code, including test generation. While many developers have adopted agent-based coding tools, little is known about how these tools generate tests in real-world development scenarios or how AI-generated tests compare to human-written ones.

Software vulnerabilities are constantly being reported and exploited in software products, causing significant impacts on society. In recent years, the main approach to vulnerability detection, fuzzing, has been integrated into the continuous integration process to run in short and frequent cycles. This continuous fuzzing allows for fast identification and remediation of vulnerabilities during the development process. Despite adoption by thousands of projects, however, it is unclear how continuous fuzzing contributes to vulnerability detection.This study aims to elucidate the role of continuous fuzzing in vulnerability detection. Specifically, we investigate the coverage and the total number of fuzzing sessions when fuzzing bugs are discovered. We collect issue reports, coverage reports, and fuzzing logs from OSS-Fuzz, an online service provided by Google that performs fuzzing during continuous integration. Through an empirical study of a total of approximately 1.12 million fuzzing sessions from 878 projects participating in OSS-Fuzz, we reveal that (i) a substantial number of fuzzing bugs exist prior to the integration of continuous fuzzing, leading to a high detection rate in the early stages; (ii) code coverage continues to increase as continuous fuzzing progresses; (iii) changes in coverage contribute to the detection of fuzzing bugs; and (iv) developer-provided seed corpus exhibit long-term effectiveness. This study provides empirical insights into how continuous fuzzing contributes to fuzzing bug detection, offering practical implications for future strategies and tool development in continuous fuzzing.

Background: Modern software systems are expected to deliver high performance under a variety of different workloads. In order to automatically verify whether a system operates correctly under specific load conditions, load testing has become a widely adopted technique. As software systems evolve, their load requirements-such as performance thresholds and usage patterns-also change, necessitating updates to load tests. Aims: This study investigates the maintenance of load testing scripts to better understand how load requirements evolve and how these changes are reflected in the tests themselves. Method: We analyzed 35 open-source software (OSS) repositories that incorporate load testing. We examined the frequency and nature of load test updates. Results: Our analysis reveals that 45.7% of the studied projects do not update their load testing scripts after initial creation. However, a small subset of projects demonstrates extensive and ongoing maintenance of these scripts. Furthermore, we identified 20 distinct update types across 5 major categories of purposes for load testing script modifications. The most frequent update type is related to “Test Maintenance”, followed by “Test Scenario Modification.” Conclusions: Our findings suggest that load testing scripts are often left unmaintained over time in many projects. The updates, when performed, serve a wide range of purposes, with test maintenance being the most frequent.

Refactoring is a widely applied practice for improving the internal structure of source code without altering its external behavior. Researchers have proposed approaches to detect refactoring operations and investigated their impact on the code quality. However, these studies often focus on production code, paying little attention to test code. It is still unclear whether developers perform refactoring on test code in the same way or for the same purpose. To fill this gap, we first investigate the types and prevalence of refactoring applied in production and test code, and then examine whether these refactorings impact the code quality in a different way. Our results show that certain refactorings are less common in the test code. Besides, while refactoring-related changes in production and/or test code improved readability, they had limited impact on most design smells. We also find that some specific refactoring types do impact certain design smells. These findings indicate the special attention needed for test code when analyzing refactorings.

Self-Admitted Technical Debt (SATD) refers to nonoptimal software design or implementation that is acknowledged and explicitly documented in the code by developers. Detecting SATD and understanding its evolution can help developers better manage their development activities and monitor the software quality. In recent years, numerous approaches have been proposed to automatically identify SATD. However, these approaches still suffer from a high number of false positives (i.e., non-SATD comments being detected as SATD). To further advance this field, in this paper, we conduct an empirical study to evaluate the performance of the state-of-theart SATD detection tools and investigate the causes behind the false positives. By manually analyzing 135 false positive cases, we identify the main types of comments that are easily misclassified. To address this issue, we propose a new approach, CASTI, which integrates context information into CodeBERT, a pre-trained model for programming languages. Our evaluation demonstrates that CASTI can significantly reduce the false positives and that the context information does help improve the performance.

Modern software development heavily relies on third-party dependencies to reduce workload and improve developer productivity. Given the vast number of dependencies available and the ease of including them in projects, some introduced dependencies are never used, leading to bloated software, longer build times, and increased network bandwidth usage. While several previous studies have examined the prevalence of unused dependencies and their impact on security, it remains unclear how these dependencies are introduced and removed in software projects. This study aims to answer this question through an empirical study involving 3,020 release versions of 417 Java projects. Our analysis shows that unused packages are common in most projects ($52 \%$ of projects), but few releases (9%) introduce new unused dependencies. Among those resolved unused dependencies, $59 \%$ of them were removed and $41 \%$ were used in later versions. Our findings highlight that not all unused dependencies should be removed in practice.

Fuzzing is an automated software testing technique used to find software vulnerabilities that works by sending large amounts of inputs to a software system to trigger bad behaviors. In recent years, the open source software ecosystem has seen a significant increase in the adoption of fuzzing to avoid spreading vulnerabilities throughout the ecosystem. While fuzzing can uncover vulnerabilities, there is currently a lack of knowledge regarding the challenges of conducting fuzzing activities over time. Specifically, fuzzers are very complex tools to set up and build before they can be used. We set out to empirically find out how challenging is build maintenance in the context of fuzzing. We mine over 1.2 million build logs from Google’s OSS-Fuzz service to investigate fuzzing build failures. We first conduct a quantitative analysis to quantify the prevalence of fuzzing build failures. We then manually investigate 677 failing fuzzing builds logs and establish a taxonomy of 25 root causes of build failures. We finally train a machine learning model to recognize common failure patterns in failing build logs. Our taxonomy can serve as a reference for practitioners conducting fuzzing build maintenance. Our modeling experiment shows the potential of using automation to simplify the process of fuzzing.

Common Vulnerability Scoring System (CVSS) is a standard method for quantifying the severity of software vulnerabilities. Security engineers can identify the underlying causes of vulnerabilities and develop countermeasures based on CVSS base vector and their related scores. The CVSS base vector is a set of metrics used to calculate the severity score of a vulnerability based on its intrinsic characteristics, such as “Attack Vector (AV)” and “Privilege Required (PR)‘. Security experts often take a few weeks to manually determine the CVSS base vector of a new vulnerability report. Consequently, the CVSS base vector is usually unknown for a period after the report is released, and thus delaying vulnerability countermeasures. Therefore, several methods for predicting CVSS base vector have been proposed. This paper compares the performance among methods that use BERT, multinomial logistic regression, or linear regression for automatically predicting CVSS base vector. The comparison results suggested that both BERT and MLR perform better than LR, with distinct advantages. BERT excels in understanding context, making it suitable for predicting general CVSS base vector, while MLR is effective for targeting specific attributes or severity levels. Consequently, these methods hold promise in aiding security engineers to promptly address vulnerabilities.

Code review is a critical but time-consuming process for ensuring code quality in modern software engineering. To alleviate the effort of reviewing source code, recent studies have investigated the possibility of automating the review process. Moreover, tools based on large language models such as ChatGPT are playing an increasingly important role in this vision. Understanding how these tools are used during code review can provide valuable insights for code review automation.

Code review plays an important role in quality assurance, which improves readability, maintainability, etc. On the other hand, code review is notorious for being very time-consuming work because reviewers need to carefully inspect numerous lines for each change. To alleviate the efforts, many studies have proposed approaches to highlighting the code that reviewers need to review. However, even using the finest-grained approaches (i.e., line-level), the granularity of the recommendations is coarse-grained so it is difficult for developers to figure out what needs to be reviewed. For example, lines in the QtBase projects have a median of 7 tokens and sometimes have more than 12 tokens (10% of the lines). This study proposes a token level approach to recommend where developers review, which is finer-grained than the previous studies. Specifically, we fine-tune CodeBERT so that it can return the tokens that are most likely to be commented on and revised. Our empirical evaluation using the OpenStack and QtBase datasets demonstrated that the proposed approach outperforms the state-of-the-art model to predict lines to be revised. Also, we find that 86% of the predicted tokens are accurately distinguished as either needing modification or not when the lines to be revised are correctly identified.

Continuous Integration (CI) provides a feedback loop for the change sets that developers produce. It is crucial that CI processes change sets quickly to provide timely feedback to developers and enable teams to release software updates rapidly. Prior work has made several advances in proposing automated approaches to speed up CI builds. While these approaches have been broadly adopted, CI platforms are flexible enough to enable teams to produce custom strategies to optimize or omit unnecessary or redundant tasks (i.e., developer-applied accelerations). Exploring developer-applied accelerations and identifying recurrent patterns within them may enable broader reuse and can inform recommendations to enhance software development efficiency.

Docker plays a crucial role in providing uniform software development. Many Docker development projects deliver multiple images in order to support various users who need different base images, versions, and architectures. To do so, the projects need to develop different contents of Dockerfiles for each support. For example, if developers provide their product on different Linux OSs, Dockerfiles need to contain package installing commands with an appropriate package manager for each Linux OS. To reduce the development tasks, many projects often develop their own tool to generate multiple Docker images automatically (hereafter, Dockerfile Preprocessors). However, it is still not clear how the projects adopt Dockerfile Preprocessors and what the benefits are. This study explores the characteristics of projects using Dockerfile Preprocessors, the timing, impact, and purpose, and the maintenance effort of using Dockerfile Preprocessors. Our empirical results show that (i) there is “Container build” pattern that does not generate multiple Dockerfiles; (ii) Projects using DPPs have more tags, supported Docker images, and architecture supports than projects without DPPs; (iii) 66% of projects develop DPPs in the middle of development; (iv) the common reasons for adopting DPP is to reduce the effort of creating Dock-erfiles, and to ease updating versions/variations/architectures; (v) the adoption of DPPs does not increase releasing activities.

Just-In-Time (JIT) defect prediction strives to model changes that induce future fixes so that they can be predicted or better understood to inform development practices. Prior work demonstrates that the majority of the predictive/explanatory power of JIT models derives from the size of a change (i.e., larger changes tend to be defect-prone); however, in practice, a misguided change to even a single line of code can lead to defects. While it is clearly the case that larger changes are more likely to alter the product behavior, even small changes are capable of doing this, and when they do, they pose a risk that teams should note. However, to the best of our knowledge, JIT defect prediction models are yet to incorporate features that characterize the change in product behavior when modelling risk. This paper is the first to explore the impact of behavioral code change on JIT prediction. Specifically, we propose seven dynamic features that capture the difference in product behavior before and after applying a change. These features are computed using trace logs that are collected during invocations of test suites. Using these logs, we identify which lines of code started/stopped being exercised after a change. We evaluate these features by conducting an empirical study of two large and thriving open-source projects. We observe that, compared to baseline models that use traditional features, adding our proposed set of behavior features leads to improvements of up to 5.9% of ROC-AVC, 44.8% of precision, and 14.1 % of PR-AUC. This paper not only demonstrates the importance of behavioral features for JIT defect prediction, but also lays the foundation for future work on behavioral features in other software engineering contexts, such as build outcome prediction and code reviewer recommendation.

Fuzz testing, also known as fuzzing, is a software testing technique aimed at identifying software vulnerabilities. In recent decades, fuzzing has gained increasing popularity in the research community. However, existing studies led by fuzzing experts mainly focus on improving the coverage and performance of fuzzing techniques. That is, there is still a gap in empirical knowledge regarding fuzzing, especially about the challenges developers face when they adopt fuzzing. Understanding these challenges can provide valuable insights to both practitioners and researchers on how to further improve fuzzing processes and techniques. We conducted a study to understand the challenges encountered by developers during fuzzing. More specifically, we first manually analyzed 829 randomly sampled fuzzing-related GitHub issues and constructed a taxonomy consisting of 39 types of challenges (22 related to the fuzzing process itself, 17 related to using external fuzzing providers). We then surveyed 106 fuzzing practitioners to verify the validity of our taxonomy and collected feedback on how the fuzzing process can be improved. Our taxonomy, accompanied with representative examples and highlighted implications, can serve as a reference point on how to better adopt fuzzing techniques for practitioners, and indicates potential directions researchers can work on toward better fuzzing approaches and practices.

Infrastructure as Code (IaC) has become the de-facto standard method for managing cloud resources. Just like general source code (e.g., Java, etc.), infrastructure code also has numerous bugs so it needs to be tested. While several testing frameworks for IaC for cloud systems have been developed in practice, researchers have paid little attention to their testing. This study presents an empirical investigation of the use of tests for IaC for cloud systems. Our empirical results show that (i) 55.2% of the repositories using Terratest have at least one server infrastructure test; (ii) developers often maintain server infrastructure tests (1.7%-11.3% commits out of all the commits); (iii) many repositories have tests for system functionality (28%), deployment (20%), and configuration (17%).

Testing is one of the most critical processes in software quality assurance. Developers spend a large portion of their time writing test code to avoid potential software failures. In recent years, snapshot testing, which compares snapshots of UI components to detect unexpected changes, has gained popularity in front-end development due to the need to reduce testing efforts. However, it is still unclear how software developers adopt snapshot testing and maintain them. To facilitate future work to reveal the potentials of snapshot testing, this paper presents a preliminary study which examines how developers use snapshot tests. More specifically, this study investigates 1) the characteristics of projects adopting snapshot testing, and 2) when snapshot tests were introduced and how they evolve. Our study is among the first to understand snapshot testing, providing valuable insights on its adoption. We also highlight the future directions to work on.

Code completion has the benefit of improving coding speed and reducing the chance of inducing bugs. In recent years, DL-based code completion techniques have been proposed. In particular, pre-trained models have shown outstanding performance because they can complete code by considering the context before and after it is completed. While the model can generate the set of candidate codes, some of those might need to be modified by developers because projects can have different coding rules.In this study, to complete code that fits a specific project appropriately, we train the CodeT5 model with additional data from the target project. This fine-tuning approach is called do-main adaptation, and is often used in neural machine translation. Our preliminary experiment observes that our domain-adapted model improves 5.3% of the perfect prediction rate and, 3.4% of the edit distance rate, compared to the fine-tuned model with the out-of-domain dataset. Furthermore, we discover that the improvement is greater with a larger repository size. The model that is trained with a small dataset, however, hardly improves or performs worse.

Background: Prior studies lack the perspective of using developer's skills to augment the performance of automated program repair (APR). APR has a phase referred to as fault localization (FL), which automatically finds the faulty statement that causes faults. To achieve a well-performed FL phase, we study developers' FL skills, which allow developers to find faulty statements. We suppose that such FL skills can add additional information to fault localization to augment the accuracy of fault localization and reduce the execution cost of APR. Aims: We aim at revealing a criterion that distinguishes whether using the FL skill reduces the execution cost of the state-of-the-art APR, TBar, depending on the accuracy of the FL skill. Method: We conduct a simulation case study in the Defects4J dataset, which is the most popular dataset. We compare the numbers of candidate patches generated by TBar using the FL skill or using spectrum-based fault localization (SBFL). Results: Our case study revealed that, if developers localized the faulty statements before inspecting 40 % of the statements in the target program, the execution cost of TBar reduces for over half of the studied faults. The 40 % value is a requirement for developers using the FL skill to augment the performance of APR. Conclusion: If developers can localize the faulty statement before inspecting 40 % of the statements, integrating the FL skill with SBFL makes TBar faster compared to when SBFL is used.

Energy consumption in mobile applications is a key area of software engineering studies, since any advance could affect billions of devices. Currently, several software-based energy calculation tools can provide close estimates of the energy consumed by mobile applications without relying on physical hardware, offering new opportunities to conduct large-scale energy studies in mobile devices. In these studies, one key step of data collection is generating events, since it allows exercising specific parts of the code and, as a consequence, assessing their energy consumption. Given the fact that manually generating events by interacting with applications is time-consuming and not scalable, large-scale studies often use software-based tools to automate event generation to profile devices. Existing tools rely on randomly generated events, which undermines the reproducibility and generalizability of such studies.We present AIP (Android Instrumentation Profiler), an alternative to existing software-based event generation tools such as Monkey. AIP uses instrumented tests as a source of event generation, which enables the targeting of complex use cases for energy consumption estimations, as well as the creation of fully reproducible events and execution traces, while maintaining the scaling abilities of other state-of-the-art tools. The tool and demo video can be found on https://github.com/ONourry/AndroidInstrumentationProfiler.

Issue reports are a pivotal interface between developers and users for receiving information about bugs in their products. In practice, issue reports often have incorrect information or insufficient information to enable bugs to be reproduced, and this has the effect of delaying the entire bug-fixing process. To facilitate their bug-reproduction work, GitHub has provided a new feature that allows users to share videos (e.g., mp4 files.) Using such videos, reports can be made to developers about the details of bugs by recording the symptoms, reproduction steps, and other important aspects of bug information.

Background: In recent years, the use of container virtualization technology has been rapidly spreading to speed up software release and operation. In general, a containerized application image (e.g., Docker image) consists of multiple reused OSS packages. To reuse OSS, it is necessary to comply with the OSS licenses. Although there have been many studies on OSS license detection and license compatibility among OSS packages, but to the best of our knowledge, there is no study tackled with incompatible license problems among OSS packages in a container image. Aims: In this paper, we conduct a preliminary analysis to clarify the extent to which Docker images contain OSS license incompatibility problems. Method: We analyze 776 Docker images published on GitHub to determine whether license incompatibilities among OSS packages exist. Results: The analysis showed that a total of 2,167 software packages were used in the 776 Docker images. The majority of the software packages (71.3%) are compatible with the GPL family, but a non-negligible number of software packages (28.7%) are not compatible. The analysis also showed that 457 (58.9%) of the 776 images had GPL-related incompatibility problems. Conclusions: Unlike traditional software development, in which software packages to be reused are explicitly combined, Dockerfile creators who build and distribute Docker images might be less aware of the risks related to compatibility between OSS licenses. Our results are useful as information to improve the awareness of Dockerfile creators, and also indicates the necessity of future studies to detect and prevent the inclusion of license-incompatible OSS packages to container images.

Abstract The accuracy of the SZZ algorithm is pivotal for just-in-time defect prediction because most prior studies have used the SZZ algorithm to detect defect-inducing commits to construct and evaluate their defect prediction models. The SZZ algorithm has two phases to detect defect-inducing commits: (1) linking issue reports in an issue-tracking system to possible defect-fixing commits in a version control system by using an issue-link algorithm (ILA); and (2) tracing the modifications of defect-fixing commits back to possible defect-inducing commits. Researchers and practitioners can address the second phase by using existing solutions such as a tool called . In contrast, although various ILAs have been proposed for the first phase, no large-scale studies exist in which such ILAs are evaluated under the same experimental conditions. Hence, we still have no conclusions regarding the best-performing ILA for the first phase. In this paper, we compare 10 ILAs collected from our systematic literature study with regards to the accuracy of detecting defect-fixing commits. In addition, we compare the defect prediction performance of ILAs and their combinations that can detect defect-fixing commits accurately. We conducted experiments on five open-source software projects. We found that all ILAs and their combinations prevented the defect prediction model from being affected by missing defect-fixing commits. In particular, the combination of a natural language text similarity approach, Phantom heuristics, a random forest approach, and a support vector machine approach is the best way to statistically significantly reduced the absolute differences from the ground-truth defect prediction performance. We summarized the guidelines to use ILAs as our recommendations.

Refactoring as a process is aimed at improving the quality of a software system while preserving its external behavior. In practice, refactoring comes in the form of many specific and diverse refactoring operations, which have different scopes and thus a different potential impact on both the production and the test code. We present a large-scale quantitative study complemented by a qualitative analysis involving 615,196 test cases to understand how and to what extent different refactoring operations impact a system's test suites. Our findings show that while the vast majority of refactoring operations do not or very seldom induce test breaks, some specific refactoring types (e.g., “RENAME Attribute” and “RENAME Class”) have a higher chance of breaking test suites. Meanwhile, “ADD Parameter” and “CHANGE Return Type” refactoring operations often require additional lines of changes to fix the test suite they break. While some modern IDEs provide features to automatically apply these two types of refactoring operations, they are not always able to avoid test breaks, thus demanding extra human efforts.

Refactoring, the process of improving the internal code structure of a software system without altering its external behavior, is widely applied during software development. Understanding how developers refactor source code can help gain better understanding of the software development process and the relationship between various versions of a system. Refactoring detection tools have been developed for many popular programming languages, such as Java (e.g., REFACTORINGMINER and REF-FINDER) but, quite surprisingly, this is not the case for Python, a widely used programming language.Inspired by REFACTORING MINER, we present PYREF, a tool that automatically detects method-level refactoring operations in Python projects. We evaluated PYREF against a manually built oracle and compared it with a PYTHON-ADAPTED REFACTOR-INGMINER, which converts Python program to Java and detects refactoring operations with REFACTORING MINER. Our results indicate that PYREF can achieve satisfactory precision and detect more refactorings than the current state-of-the-art.

Over a decade, many bug assignment methods have been proposed in order to assist developers to read bug reports submitted daily and numerously, and to assign an appropriate developer. However, they tend to concentrate their assignments on a small number of particular developers. Applying the methods to the projects which have releases would reduce the number of bugs that developers can fix by the next release date because the time that developers can devote to bug-fixing is limited. In this study, we propose the release-aware bug-fixing task assignment method to mitigate the task concentration and increase the number of bugs that developers can fix by the next release date. This method employs mathematical programming to find the best combination at project level while the traditional methods find the best pair of a bug and a developer (at individual level).

The importance of supporting test and maintenance activities in software development has been increasing, since recent software systems have become large and complex. Although in the field of Mining Software Repositories (MSR) there are many promising approaches to predicting, localizing, and triaging bugs, most of them do not consider impacts of each bug on users and developers but rather treat all bugs with equal weighting, excepting a few studies on high impact bugs including security, performance, blocking, and so forth. To make MSR techniques more actionable and effective in practice, we need deeper understandings of high impact bugs. In this paper we introduced our dataset of high impact bugs which was created by manually reviewing four thousand issue reports in four open source projects (Ambari, Camel, Derby and Wicket).

Since increasing complexity and scale of modern software products imposes tight scheduling and resource allocations on software development projects, a project manager must carefully triage bugs to determine which bug should be necessarily fixed before shipping. Although in the field of Mining Software Repositories (MSR) there are many promising approaches to predicting, localizing, and triaging bugs, most of them do not consider impacts of each bug on users and developers but rather treat all bugs with equal weighting, excepting a few studies on high impact bugs including security, performance, blocking, and so forth. To make MSR techniques more actionable and effective in practice, we need deeper understandings of high impact bugs. In this paper we report our pilot study on high impact bugs, which classifies bugs reported to four open source projects into six types of high impact bugs.