Traceability links are key information sources for software developers, connecting software artifacts. Such links play an important role, particularly between contribution artifacts and their corresponding source code. Through these links, developers can trace the discussions in contributions and uncover design rationales, constraints, and security concerns. Previous studies have mainly examined accepted contributions, while those declined after discussion have been overlooked. Declined-contribution discussions capture valuable design rationale and implicit decision criteria, revealing why features are accepted or rejected. Our prior work also shows developers often revisit and resubmit declined contributions, making traceability to them useful.

Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. In this study, we investigate to what extent large language models (LLMs) can generate SATD taxonomies. We designed a structured, LLM-driven pipeline that mirrors the taxonomy construction steps researchers typically follow. We evaluated it on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovered domain-specific categories reported in prior work, such as Layer Configuration in machine learning. It also completed taxonomy generation in under two hours and for less than ${\$}$1, even on the largest dataset. These results suggest that, while full automation remains challenging, LLMs can support semi-automated SATD taxonomy construction. Furthermore, our work opens up avenues for future work, such as automated taxonomy generation in other areas.

Code review is a key practice in software engineering, where developers evaluate code changes to ensure quality and maintainability. Links to issues and external resources are often included in Pull Requests (PRs) to provide additional context, yet they are typically discarded in automated tasks such as PR summarization and code review comment generation. This limits the richness of information available to reviewers and increases cognitive load by forcing context-switching. To address this gap, we present AILinkPreviewer, a tool that leverages Large Language Models (LLMs) to generate previews of links in PRs using PR metadata, including titles, descriptions, comments, and link body content. We analyzed 50 engineered GitHub repositories and compared three approaches: Contextual LLM summaries, Non-Contextual LLM summaries, and Metadata-based previews. The results in metrics such as BLEU, BERTScore, and compression ratio show that contextual summaries consistently outperform other methods. However, in a user study with seven participants, most preferred non-contextual summaries, suggesting a trade-off between metric performance and perceived usability. These findings demonstrate the potential of LLM-powered link previews to enhance code review efficiency and to provide richer context for developers and automation in software engineering. The video demo is available at https://www.youtube.com/ watch?v $=h 2 q \mathrm{H} 4 \mathrm{R} t r B 3 \mathrm{E}$, and the tool and its source code can be found at https://github.com/c4rtune/AILinkPreviewer.

Quantum computers leverage the principles of quantum mechanics to execute operations. They require quantum programs that define operations on quantum bits (qubits), the fundamental units of computation. Unlike traditional software development, the process of creating and debugging quantum programs requires specialized knowledge of quantum computation, making the development process more challenging.

Background: Large language models (LLMs) have greatly improved the accuracy of automated program repair (APR) methods. However, LLMs are constrained by high computational resource requirements. Aims: We focus on small language models (SLMs), which perform well even with limited computational resources compared to LLMs. We aim to evaluate whether SLMs can achieve competitive performance in APR tasks. Method: We conducted experiments on the QuixBugs benchmark to compare the bug-fixing accuracy of SLMs and LLMs. We also analyzed the impact of int8 quantization on APR performance. Results: The latest SLMs can fix bugs as accurately as-or even more accurately than-LLMs. Also, int8 quantization had minimal effect on APR accuracy while significantly reducing memory requirements. Conclusions: SLMs present a viable alternative to LLMs for APR, offering competitive accuracy with lower computational costs, and quantization can further enhance their efficiency without compromising effectiveness.

Large Language Models (LLMs) have shown great potential in code-related software engineering tasks, including code generation, classification, and understanding. While current research primarily focuses on direct inference with single LLMs, this approach may fall short for complex tasks due to ambiguous instructions. To address this limitation, we propose LLM selfnegotiation, where multiple LLMs collaborate and debate to reach consensus on code-related tasks. This approach aims to better handle unclear instructions and improve overall effectiveness. We evaluated LLM self-negotiation in three key software engineering domains: Equivalent Mutant Detection (EMD), Automated Vulnerability Detection (AVD), and Automated Program Repair (APR). These domains represent distinct aspects of coderelated tasks: functionality understanding, code classification, and code generation, respectively. Our experimental results revealed varying effectiveness across domains. In EMD, LLM self-negotiation demonstrated remarkable improvements, with most models showing performance gains between 114.72 % and 351.01% (though CodeLlama experienced a minor 4.5% decrease in F1-score). For APR tasks, self-negotiation performed comparably to single LLM implementations. However, in AVD, the results were mixed - while Vicuna showed improved F1-scores, most models exhibited lower recall rates. These findings indicate that LLM self-negotiation is particularly promising for functionality understanding tasks, while its application to code classification and generation requires further research and refinement.

With the increasing prevalence of software incorporating deep neural networks (DNNs), quality assurance for these software systems has become a crucial concern. To this end, various methods have been proposed to repair the misbehavior of DNNs by modifying their weights. However, these repair methods may not meet the developer’s needs for a given dataset and model. In this study, we build prediction models for repair outcomes (i.e., repairs and breaks) to help determine whether the repair method is likely to work. By using our prediction models, developers and operators of DNNs can decide whether or not to apply a repair method, and if so, which method to use. Our prediction models utilize four metrics as explanatory metrics that represent the confidence or ambiguity in the DNN predictions. We experimented with four repair methods and 10 datasets. The experimental results demonstrate that our prediction models successfully select a repair method that meets developers’ needs in 16 out of 24 cases, resulting in an average time saving of 16.29% compared to the naive method. Based on these results, our prediction models can reduce costs for developers and operators when deciding whether to employ repair methods for real-world applications of DNNs.

Quantum computers, which utilize the principles of quantum mechanics, are expected to be applied to a wide range of fields. With the advancement of quantum computer development, a lot of quantum software, which enables the operation of quantum computers, has been developed. It has a distinct nature (e.g., superposition and entanglement of qubits) compared to traditional software, leading to the unique challenges of its development. While prior studies have clarified and defined some unique challenges of quantum software, many remain unclear due to limited research. In this study, we conducted an empirical study of Self-Admitted Technical Debt (SATD) for quantum software. SATD is a type of technical debt, a problem in the code that the developer is aware of. Hence, we conjecture that analyzing SATDs can reveal the unique challenges developers face when developing quantum software. We manually coded 202 comments from the Python® files of the 61 open-source quantum software on GitHub®. The 202 comments correspond to a 95% confidence level with a 5% confidence interval, as in previous studies. The results showed that 88 comments (45.6% of all SATD comments) were quantum-specific SATDs (QSATDs), which require knowledge of quantum computation to repay. Furthermore, we propose a taxonomy for QSATDs. This taxonomy, which consists of four main categories and eight subcategories, classifies QSATDs in terms of quantum-specific aspects such as circuit implementation, backend, and algorithms. Our empirical results are beneficial for quantum software developers, helping them understand implementation areas that require attention. For researchers, our results promote further research, including the exploration of challenges in QSATD repayment.

Fuzz testing (or fuzzing) is a software testing technique aimed at identifying software vulnerabilities. Recently, the Go community added native support for fuzz testing into their standard library. Using that feature, developers can write unit tests to perform deterministic and fuzz testing of their software systems against unexpected inputs. Although the availability of support makes fuzz testing more accessible for the Go community at large, little is known about the degree to which Go developers adopt fuzz testing during software development. Therefore, in this paper, we set out to study the evolution of fuzz testing practices in open-source Go projects. More specifically, we strive to understand whether the introduction of support for fuzz testing in the Go standard library has led to the adoption of fuzz testing as part of the standard testing processes of Go projects. To achieve our goal, we study 1) to what extent fuzz tests are used in open-source Go projects, 2) who writes and maintains fuzz tests in Go projects, and finally, 3) how tightly coupled are fuzz tests with source code (as compared to non-fuzz tests). We find that fuzz testing only represents 3.15% of testing functions in open-source projects. Our results also suggest that fuzz testing development is not being conducted as part of standard testing activities. For developers contributing to fuzzing, we find that a median of only 12.50% of their testing-related commits contain fuzz tests. Finally, we perform a qualitative analysis and find that fuzz testing is mostly used by critical software systems, such as blockchain technologies or network infrastructure projects, to test the most critical features of their systems (e.g., data processing functions, database endpoints). Our results lead us to conclude that fuzz testing is best used in combination with deterministic testing (e.g., unit testing) where fuzzing is used to thoroughly test important features, and deterministic testing is used to test other features.

GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors.

Just-In-Time (JIT) defect prediction strives to model changes that induce future fixes so that they can be predicted or better understood to inform development practices. Prior work demonstrates that the majority of the predictive/explanatory power of JIT models derives from the size of a change (i.e., larger changes tend to be defect-prone); however, in practice, a misguided change to even a single line of code can lead to defects. While it is clearly the case that larger changes are more likely to alter the product behavior, even small changes are capable of doing this, and when they do, they pose a risk that teams should note. However, to the best of our knowledge, JIT defect prediction models are yet to incorporate features that characterize the change in product behavior when modelling risk. This paper is the first to explore the impact of behavioral code change on JIT prediction. Specifically, we propose seven dynamic features that capture the difference in product behavior before and after applying a change. These features are computed using trace logs that are collected during invocations of test suites. Using these logs, we identify which lines of code started/stopped being exercised after a change. We evaluate these features by conducting an empirical study of two large and thriving open-source projects. We observe that, compared to baseline models that use traditional features, adding our proposed set of behavior features leads to improvements of up to 5.9% of ROC-AVC, 44.8% of precision, and 14.1 % of PR-AUC. This paper not only demonstrates the importance of behavioral features for JIT defect prediction, but also lays the foundation for future work on behavioral features in other software engineering contexts, such as build outcome prediction and code reviewer recommendation.

SZZ is the de facto standard method for identifying bug-inducing commits. The accuracy of this method heavily relies on source code management systems, such as Git, as it requires tracing the history of source code changes (i.e., commit histories) to bug-inducing commits. However, it has been reported that these systems introduce biases in commit histories because they only store line-level changes. It is known that such coarse-grained line-level changes can result in the failure to accurately track the commit history and reduce the performance of SZZ. To relieve this challenge, we explore the accuracy of SZZ in token-level changes, which provide finer-grained information to trace commit histories compared to line-level ones, and we discuss the potential benefits and pitfalls of utilizing token-level changes for SZZ. As a result of experiments on 68 OSS projects, we found that SZZ, which uses token-level histories, identifies two new bug-inducing commits that are missed when using line-level histories. Furthermore, our manual analysis of the identified commits indicates that they reduce false-positive bug-inducing commits caused by source code formatting and whitespace changes. However, this improvement in detecting bug-inducing commits comes with a trade-off of 0.081 decrease in overall accuracy, as measured by the F1 score. Consequently, we summarized three potential benefits and five pitfalls of using token-level and line-level tracking for SZZ.

With the prevalence of software systems adopting neural network models, the quality assurance of these systems has become crucial. Hence, various studies have proposed repairing methods for neural network models so far to improve the quality of the models. While these methods are evaluated by researchers, it is difficult to tell whether they succeed in all models and datasets (i.e., all developers’ environments). Because these methods require many resources, such as execution times, failing to repair neural networks would cost developers their resources. Hence, if developers can know whether repairing methods succeed before adopting them, they could avoid wasting their resources. This paper proposes prediction models that predict whether repairing methods succeed in repairing neural networks using a small resource. Our prediction models predict repairs and side-effects of repairing methods, respectively. We evaluated our prediction models on a state-of-the-art repairing method Arachne on three datasets, Fashion-MNIST, CIFAR-10, and GTSRB, and found our prediction models achieved high performance, an average ROC-AUC of 0.931 and an average f1score of 0.880 for the side-effects and an average ROC-AUC of 0.768 and an average f1-score of 0.725 for the repairs.

COVID-19 has transformed face-to-face software development into distributed development (e. g., remote work). While the company authors belong to studies microtask programming, an open source software (OSS) -like development, as a solution to employ distributed development, a prior study reports a challenge: online communication in microtask programming takes longer; such lengthy communication discourages developers and affects their completion of assigned tasks. OSS, however, is successfully developed using online communication, such as issues. Hence, we have a question: how does OSS address the online communication challenge? In this experience report, we answer this question based on an empirical study on OSS communication. We found that (1) OSS prefers burst communication similar to face-to-face development, and (2) attracting developers’ attention may be a possible solution. Based on the findings, we discuss the direction of future studies to achieve better online communication in microtask programming in the company. The main contributions of this report are (1) to empirically reveal the actual communication times in OSS and (2) to show how an empirical approach helps industrial collaborators.

Defect prediction models can predict defects in software projects, and many researchers study defect prediction models to assist debugging efforts in software development. In recent years, there has been growing interest in Cross Project Defect Prediction (CPDP), which predicts defects in a project using a defect prediction model learned from other projects’ data when there is insufficient data to construct a defect prediction model. Since CPDP uses other projects’ data, data privacy preservation is one of the most significant issues. However, prior CPDP studies still require data sharing among projects to train models, and do not fully consider protecting project confidentiality. To address this, we propose a CPDP model FLR employing federated learning, a distributed machine learning approach that does not require data sharing. We evaluate FLR, using 25 projects, to investigate its effectiveness and feature interpretation. Our key results show that first, FLR outperforms the existing privacy-preserving methods (i.e., LACE2). Meanwhile, the performance is relatively comparable to the conventional methods (e.g., supervised and unsupervised learning). Second, the results of the interpretation analysis show that scale-related features have a common effect on the prediction performance of the FLR. In addition, further insights demonstrate that parameters of federated learning (e.g., learning rates and the number of clients) also play a role in the performance. This study is served as a first step to confirm the feasibility of the employment of federated learning in CPDP to ensure privacy preservation and lays the groundwork for future research on applying other machine learning models to federated learning.

If deep learning models in safety–critical systems misbehave, serious accidents may occur. Previous studies have proposed approaches to overcome such misbehavior by detecting and modifying the responsible faulty parts in deep learning models. For example, fault localization has been applied to deep neural networks to detect neurons that cause misbehavior. However, such approaches are not applicable to deep learning models that have internal states, which change dynamically based on the input data samples (e.g., recurrent neural networks (RNNs)). Hence, we propose a new fault localization approach to be applied to RNNs. We propose probabilistic automaton-based fault localization (PAFL). PAFL enables developers to detect faulty parts even in RNNs by computing suspiciousness scores with fault localization using n-grams. We convert RNNs into probabilistic finite automata (PFAs) and localize faulty sequences of state transitions on PFAs. To consider various sequences and to detect faulty ones more precisely, we use n-grams inspired by natural language processing. Additionally, we distinguish data samples related to the misbehavior to evaluate PAFL. We also propose a novel suspiciousness score, average n-gram suspiciousness (ANS) score, based on n-grams to distinguish data samples. We evaluate PAFL and ANS scores on eight publicly available datasets on three RNN variants: simple recurrent neural network, gated recurrent units, and long short-term memory. The experiment demonstrates that ANS scores identify faulty parts of RNNs when n is greater than one. Moreover, PAFL is statistically significantly better and has large effect sizes compared to state-of-the-art fault localization in terms of distinguishing data samples related to the misbehavior. Specifically, PAFL is better in 66.74% of the experimental settings. The results demonstrate that PAFL can be used to detect faulty parts in RNNs. Hence, in future studies, PAFL can be used as a baseline for fault localization in RNNs.

Debugging erroneous programs requires a great deal of human effort. To reduce human effort, automating debugging processes has been actively studied so far. One of such automation is automated program repair techniques for syntactic errors in programs. Researchers intend to support novice programmers such as students with these techniques because fixing syntactic errors is a difficult task for novice programmers. However, there exist few datasets that consist of programs written by novice programmers in universities and can be used to evaluate these techniques. Also, it is difficult to prepare such datasets from scratch. Indeed, prior studies usually utilized the Indian Institute of Technology Kanpur (IITK) dataset only. This limitation restricts the findings and implications in prior studies as a case study in the university. In this study, we intend to clarify which findings and implications in prior studies remain the same and which ones change in another university by a case study. We prepare three datasets that consist of over 21k programs collected from an introductory programming course in different divisions at our university. We compare the state-of-the-art automated program repair techniques, DeepFix, RLAssist, and Dr Repair, in these datasets. We found that (1) the best technique remains the same in all the datasets, (2) these techniques fix 8.3 % to 54.5 % syntactic errors in our datasets, which are 6.7 % to 32.4 % lower than those in the IITK dataset, (3) the error types that are fixed by them change in different datasets. Hence, the main finding in the IITK dataset remains the same; however, each technique fixes different errors.

Background: Prior studies lack the perspective of using developer's skills to augment the performance of automated program repair (APR). APR has a phase referred to as fault localization (FL), which automatically finds the faulty statement that causes faults. To achieve a well-performed FL phase, we study developers' FL skills, which allow developers to find faulty statements. We suppose that such FL skills can add additional information to fault localization to augment the accuracy of fault localization and reduce the execution cost of APR. Aims: We aim at revealing a criterion that distinguishes whether using the FL skill reduces the execution cost of the state-of-the-art APR, TBar, depending on the accuracy of the FL skill. Method: We conduct a simulation case study in the Defects4J dataset, which is the most popular dataset. We compare the numbers of candidate patches generated by TBar using the FL skill or using spectrum-based fault localization (SBFL). Results: Our case study revealed that, if developers localized the faulty statements before inspecting 40 % of the statements in the target program, the execution cost of TBar reduces for over half of the studied faults. The 40 % value is a requirement for developers using the FL skill to augment the performance of APR. Conclusion: If developers can localize the faulty statement before inspecting 40 % of the statements, integrating the FL skill with SBFL makes TBar faster compared to when SBFL is used.

Issue reports are a pivotal interface between developers and users for receiving information about bugs in their products. In practice, issue reports often have incorrect information or insufficient information to enable bugs to be reproduced, and this has the effect of delaying the entire bug-fixing process. To facilitate their bug-reproduction work, GitHub has provided a new feature that allows users to share videos (e.g., mp4 files.) Using such videos, reports can be made to developers about the details of bugs by recording the symptoms, reproduction steps, and other important aspects of bug information.

Microtask programming [4] is a solution to promote distributed development in industry. The key idea of microtask programming is to reduce face-to-face communication across developers by splitting the development task of software into independent microtasks. Such microtasks can be completed by crowd workers who work remotely and at their preferable time such as early morning. Dedicated developers who have the responsibility for the progress of development split the task into microtasks, and distribute them to crowd workers. Hence, microtask programming has these two actors. Our research team reported that microtask programming has potential benefits such as the fluidity of project assignments in industrial companies [4]. However, we suppose it still has challenges. In addition, it is still unclear what are future research direction to support both actors in microtask programming, though our research team has conducted three studies for microtask programming so far [2--4].

Abstract The accuracy of the SZZ algorithm is pivotal for just-in-time defect prediction because most prior studies have used the SZZ algorithm to detect defect-inducing commits to construct and evaluate their defect prediction models. The SZZ algorithm has two phases to detect defect-inducing commits: (1) linking issue reports in an issue-tracking system to possible defect-fixing commits in a version control system by using an issue-link algorithm (ILA); and (2) tracing the modifications of defect-fixing commits back to possible defect-inducing commits. Researchers and practitioners can address the second phase by using existing solutions such as a tool called . In contrast, although various ILAs have been proposed for the first phase, no large-scale studies exist in which such ILAs are evaluated under the same experimental conditions. Hence, we still have no conclusions regarding the best-performing ILA for the first phase. In this paper, we compare 10 ILAs collected from our systematic literature study with regards to the accuracy of detecting defect-fixing commits. In addition, we compare the defect prediction performance of ILAs and their combinations that can detect defect-fixing commits accurately. We conducted experiments on five open-source software projects. We found that all ILAs and their combinations prevented the defect prediction model from being affected by missing defect-fixing commits. In particular, the combination of a natural language text similarity approach, Phantom heuristics, a random forest approach, and a support vector machine approach is the best way to statistically significantly reduced the absolute differences from the ground-truth defect prediction performance. We summarized the guidelines to use ILAs as our recommendations.

In recent years, a configuration management tool is adopted to manage complicated and huge systems such as bare-metal servers, cloud computing resources and our personal computers. Such a tool makes the operations to deploy services more efficient and eliminates dependencies on the specific system operators. The operations are required to be idempotent for reproducible deployment. However, the imperative modules whose operations may not be idempotent are used frequently to execute user-defined scripts on the target system; it is unclear why and how they are used, though using them frequently is believed to be a bad practice. In this paper, we studied why and how imperative modules are used in a configuration management tool, Ansible. We found that imperative modules are mainly used to perform operations that are not supported by Ansible, and about 45% of imperative modules are replaceable by other modules; the replaceable modules might be idempotent. We, therefore, recommend developers to look at replaceable modules before using imperative modules since replaceable modules might make their operations idempotent.

GitHub is a huge publicly available development platform for hosting a version control system based on Git; software developers prefer to host their various software projects in GitHub. Therefore researchers who are interested in mining software repository frequently use GitHub to collect software projects as datasets. GitHub provides us with repository metrics such as popularity, contribution, and interest. We believe that such metrics are related to the quality of software; we use them to opt for studied repositories according to our research purpose. However, to the best of our knowledge, nobody has any evidence to support this assumption.Our main purpose is to provide researchers who study software quality, especially issue management, with repository metrics to select appropriate repositories for their studies. In this paper, we study the relationship between the characteristics of the issue pages of repositories that are selected by repository metrics in order to figure out the best repository metric to select proper repositories. The following findings are the highlights of our study: (1) The number of contributors that indicates the number of developers who contribute to a GitHub repository can be used to select the repositories having issue pages that are well-maintained. More specifically, such issue pages include more issues and in which developers use the labels more frequently rather than those that are selected by other metrics. (2) The number of dependencies opts for the repositories that have fewer issues and in which developers use the labels less often rather than those that are selected by other metrics.