Self-Admitted Technical Debt (SATD) refers to sub-optimal solutions deliberately introduced to accelerate the software development process, often at the expense of software maintainability and sustainability. Therefore, timely identification and repayment of the SATD is critical for the software system. As exploration deepens, it is found that effectively prioritizing the repayment of SATD with more significant impacts on software quality requires not only identifying SATD but also further classifying it. However, existing SATD identification and classification approaches face the following challenges: (1) SATDs originate from diverse sources. Code comments are a widespread source, but recent research has revealed that SATDs can originate from other sources, such as pull requests, issues, and commit messages. Nonetheless, existing approaches primarily target code comments, lacking the capability to analyze SATDs from other sources effectively. (2) SATDs fall into diverse categories. Nonetheless, existing SATD classification approaches fail to address all SATD categories comprehensively and show inadequate performance. (3) Imbalance of existing SATD datasets. Real-world SATD data are scarce, making dataset collection challenging. Moreover, SATD distribution across different sources is uneven, further complicating the construction of high-quality datasets. To alleviate these challenges, this article presents an SATD identification and classification framework named IMPACT . First, IMPACT employs ChatGPT to construct an augmented dataset. Subsequently, it utilizes a pipeline with two fine-tuned language models of different parameter sizes to identify and classify SATD separately. To evaluate the effectiveness of IMPACT, we compare it with three state-of-the-art SATD classification methods and its two foundation models. Experimental results demonstrate that IMPACT outperforms state-of-the-art methods by a large margin, and even surpasses its foundation model GLM-4-9B-Chat. It achieves the optimal average F1 score of 0.697 on the source of pull requests, the most challenging data source. Moreover, experiments on the cross-project test set show that IMPACT demonstrates strong generalizability on unseen project data.

Recent years have seen a rise in Neural Program Repair (NPR) systems in the software engineering community, which adopt advanced deep learning techniques to automatically fix bugs. Having a comprehensive understanding of existing systems can facilitate new improvements in this area and provide practical instructions for users. However, we observe two potential weaknesses in the current evaluation of NPR systems: ① published systems are trained with varying data, and ② NPR systems are roughly evaluated through the number of totally fixed bugs. Questions such as what types of bugs are repairable for current systems cannot be answered yet. Consequently, researchers cannot make target improvements in this area and users have no idea of the real affair of existing systems. In this article, we perform a systematic evaluation of the existing nine state-of-the-art NPR systems. To perform a fair and detailed comparison, we (1) build a new benchmark and framework that supports training and validating the nine systems with unified data and (2) evaluate re-trained systems with detailed performance analysis, especially on the effectiveness and the efficiency. We believe our benchmark tool and evaluation results could offer practitioners the real affairs of current NPR systems and the implications of further facilitating the improvements of NPR.

To date, over 40 Automated Program Repair (APR) tools have been designed with varying bug-fixing strategies, which have been demonstrated to have complementary performance in terms of being effective for different bug classes. Intuitively, it should be feasible to improve the overall bug-fixing performance of APR via assembling existing tools. Unfortunately, simply invoking all available APR tools for a given bug can result in unacceptable costs on APR execution as well as on patch validation (via expensive testing). Therefore, while assembling existing tools is appealing, it requires an efficient strategy to reconcile the need to fix more bugs and the requirements for practicality. In light of this problem, we propose a Preference-based Ensemble Program Repair framework (P-EPR), which seeks to effectively rank APR tools for repairing different bugs. P-EPR is the first non-learning-based APR ensemble method that is novel in its exploitation of repair patterns as a major source of knowledge for ranking APR tools and its reliance on a dynamic update strategy that enables it to immediately exploit and benefit from newly derived repair results. Experimental results show that P-EPR outperforms existing strategies significantly both in flexibility and effectiveness.

Abstract Due to the high cost of repairing defective programs, many researches focus on automatic program repair (APR). In recent years, the new trend of APR is to apply neural networks to mine the relations between defective programs and corresponding patches automatically, which is known as neural program repair (NPR). The community, however, ignores some important properties that could impact the applicability of NPR systems, such as robustness. For semantic‐identical buggy programs, NPR systems may produce totally different patches. In this paper, we propose an evaluation tool named RobustNPR, the first NPR robustness evaluation tool. RobustNPR employs several mutators to generate semantic‐identical mutants of defective programs. For an original defective program and its mutant, it checks two aspects of NPR: (a) Can NPR fix mutants when it can fix the original defective program? and (b) can NPR generate semantic‐identical patches for the original program and the mutant? Then, we evaluate four SOTA NPR models and analyze the results. From the results, we find that even for the best‐performing model, 20.16% of the repair success is unreliable, which indicates that the robustness of NPR is not perfect. In addition, we find that the robustness of NPR is correlated with model settings and other factors.

Developing effective data-driven automated bug-fixing approaches is heavily relying on large bug-fix datasets. However, the granularity of current repository-mined bug-fixing datasets is usually at the function level, without meta-information such as the fault type. In order to alleviate the open challenge of precisely mining code snippets with bugs, their fix, location, and types from open source repositories, in this paper, we propose a flexible, extensible, and automated multilingual bug-fix dataset construction system, that is, the Multilingual Bug-Fix Constructor (MBFC). Furthermore, we release a large-scale and fine-grained Multi-lingual Bug-Fix (M-BF) dataset automatically built using the proposed system, which includes a total of 921,825 Bug-Fix pairs that are from 442,164 different open-source software projects starting from January 2020 to September 2020 in the initial version. It is expected that our system and dataset can benefit the development of innovative and practical program repair methods, thereby improving the efficiency of program debugging and code review processes.

Automated Program Repair (APR) aims to automatically fix bugs in the source code. Recently, with advances in Deep Learning (DL) field, there has been an increase of Neural Program Repair (NPR) studies that use neural networks to model the patch-generation process. NPR approaches have a significant benefit in applicability over prior APR techniques because they do not require any specifications (e.g., a test suite) when generating patches. For this reason, NPR has recently become a popular research topic. In this paper, We undertake a literature review of latest NPR systems to help interested readers understand advancements in this emerging field. We begin by introducing background information of NPR. Next, to make the various NPR systems more understandable, we split them into a four-phase pipeline and discuss various design choices for each phase. To investigate the motivations of different design choices, We further highlight a number of challenges and summarize corresponding solutions adopted by existing NPR systems. Finally, we suggest some intriguing directions for the future research.

Recently, the emerging trend in automatic program repair is to apply deep neural networks to generate fixed code from buggy ones, called NPR (Neural Program Repair). However, the existing NPR systems are trained and evaluated under very different settings (e.g., different training data, inconsistent evaluation data, wide-ranged candidate numbers), which makes it hard to draw fair-enough conclusions when comparing them. Motivated by this, we first build a standard benchmark dataset and an extensive framework tool to mitigate threats for the comparison. The dataset consists of a training set, a validation set and an evaluation set with 144,641, 13,739 and 13,706 bug-fix pairs of Java respectively. The tool supports selecting specific training, validation, and evaluation datasets and automatically conducting the pipeline of training and evaluating NPR models, as well as easily integrating new NPR models by implementing well-defined interfaces. Then, based on the benchmark and tool, we conduct a comprehensive empirical comparison of six SOTA NPR systems w.r.t the repairability, inclination and generalizability. The experimental results reveal deeper characteristics of compared NPR systems and subvert some existing comparative conclusions, which further verify the necessity of unifying the experimental setups in exploring the progresses of NPR systems. Meanwhile, we reveal some common features of NPR systems (e.g., they are good at dealing with code-delete bugs). Finally, we identify some promising research directions derived from our findings.