Continuous Integration (CI) is a fundamental practice in modern software development. It enables early fault detection through regression testing, where Test Case Prioritization (TCP) plays a crucial role in improving detection efficiency. Reinforcement Learning-based TCP (RL-TCP) has shown promise in CI scenarios, but its performance often fluctuates due to CI’s dynamic nature. Existing solutions address this by assigning additional rewards or periodically retraining agents. However, these methods either risk over-adjusting strategies due to excessive reliance on additional rewards or fail to respond promptly to changes due to fixed retraining intervals. We propose a novel Environment Adaptation Agent-based RL-TCP method (EAA) that addresses these challenges through a dual mechanism. EAA detects significant environmental changes by analyzing fluctuations in prioritization effectiveness. When a change is detected, it assigns targeted rewards to test cases. EAA also refines the agent’s gradient update so that environmental dynamics are better incorporated into retraining. This enables agents to swiftly adapt while retaining learned prioritization knowledge. Evaluations on 12 real-world industrial datasets show that EAA improves the NAPFD metric by 4.7–24.79% and reduces the average TTF by 35.85–50.37 positions compared to state-of-the-art RL-TCP methods. Additionally, EAA significantly reduces occurrences of NAPFD equal to zero, effectively mitigating prioritization instability.

Fault localization (FL) in novice programs is critical for computer science education, yet it remains insufficiently explored compared to industrial programs. Traditional FL methods such as Spectrum-Based Fault Localization (SBFL) and Mutation-Based Fault Localization (MBFL) primarily rely on test coverage or mutation analysis, exhibiting significant limitations when handling novice programs. In recent years, Large Language Models (LLMs) have enabled more effective fault localization through semantic code analysis. However, existing LLMs primarily rely on internal knowledge acquired during training and lack access to specialized external knowledge bases, particularly limiting their effectiveness when addressing novice programs with distinctive fault patterns. To address these limitations, we propose NPFL-RAG, a LLM-based fault localization framework that combines Retrieval Augmented Generation (RAG). This framework regards the FL task as a three-step process: Knowledge Base Construction, which builds a comprehensive repository containing multidimensional fault-fixing information; Relevant Fix Cases Identification, which employs multidimensional retrieval and Retrieval Results Optimization (RRO) mechanism to identify the most relevant fix cases; and Fault Localization Result Generation, which integrates the retrieved knowledge with the LLM's inherent understanding to localize potential faults. We conduct a comprehensive experimental evaluation on the TutorCode dataset. The results demonstrate that NPFL-RAG significantly outperforms baselines across multiple evaluation metrics. In particular, when incorporating the RAG mechanism, DeepSeek-v3 achieves a 7.1 % improvement in the TOP-1 metric compared to the without RAG technique and performs approximately 7 times better than traditional MBFL techniques. Furthermore, we confirm the indispensability of the components in NPFL-RAG with the ablation study and demonstrate the usability of RRO mechanism in fault localization and explanation tasks.

Large Language Models (LLMs) can pose serious risks in real-world applications due to their potential for erroneous behavior, necessitating comprehensive and effective testing of LLMs. To assess the robustness of LLMs, adversarial attacks are typically conducted by constructing adversarial examples. Previous methods often require extensive queries and access to the internal information of the victim model. However, the internal information of most black-box LLMs is not accessible, rendering these testing methods infeasible. In addition, excessive queries to commercial black-box LLMs may incur substantial costs. To address these issues, this paper proposes an E nsemble-based T ransfer T esting method for L arge L anguage M odels (ETTLLM). In contrast to previous adversarial testing methods for LLMs, ETTLLM queries white-box surrogates rather than the victim model, thereby significantly reducing testing costs. Moreover, it enhances the transferability and generalization of adversarial examples across diverse real-world classification tasks. Compared to baselines, ETTLLM significantly reduced the number of queries to the victim model, with an average of 1.6 queries, just 1.2% of the baselines. Furthermore, the textual similarity and modification rate of the adversarial examples generated by ETTLLM differ from the baselines by no more than 1.6%, while achieving 70% of the attack success rate compared to the baselines.

In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).

Code generation has gained increasing attention as a task to automate software development by transforming high-level descriptions into executable code. While large language models (LLMs) are effective in generating code, their performance heavily relies on the quality of input prompts. Current prompt engineering methods involve manual effort in designing prompts, which can be time-consuming and yield inconsistent results, potentially constraining the efficacy of LLMs in practical applications. This paper introduces Prochemy, a novel approach for automatically refining prompts iteratively to enhance code generation. Prochemy addresses the limitations of manual prompt engineering by automating the optimization process, ensuring prompt consistency during inference, and aligning with multi-agent systems. It iteratively refines prompts based on model performance, using an optimized final prompt to improve consistency and reliability across tasks. We evaluate Prochemy on both natural language-based code generation and code translation tasks using three series of LLMs. Results show that when combining Prochemy with existing approaches, it outperforms baseline prompting methods. It achieves improvements of 5.0% (GPT-3.5-Turbo) and 1.9% (GPT-4o) over zero-shot baselines on HumanEval. For the state-of-the-art LDB, Prochemy + LDB outperforms standalone methods by 1.2–1.8%. For code translation, Prochemy elevates GPT-4o’s performance on Java-to-Python (AVATAR) from 74.5 to 84.1 (+12.9%) and Python-to-Java from 66.8 to 78.2 (+17.1%). Furthermore, considering that the o1-mini model integrates prompt engineering techniques, Prochemy can continue to show good performance among it, further validating its effectiveness in code generation and translation tasks. Additionally, Prochemy is designed to be plug-and-play, optimizing prompts with minimal human intervention and seamlessly bridging the gap between simple prompts and complex frameworks.

As a key phase in software testing and debugging, fault localization can significantly influence the efficiency of fixing software faults. Among the various techniques, Mutation-Based Fault Localization (MBFL) is a widely studied fault localization technique that uses mutation analysis to guide the process of localizing faults. However, as the essential input source for MBFL, traditional mutation generates syntactical mutants, which cannot mimic the real faults and may affect the fault localization effectiveness. To address this issue, we resort to a code pre-trained model for program mutation, which is called neural mutation. Neural mutation can generate semantical mutants and even utilize the context information surrounding the mutation position. Based on the neural mutation, we propose Neural-MBFL by utilizing the high-quality mutants generated by neural mutation. To evaluate the effectiveness of Neural- MBFL, we conduct experiments on 393 faulty programs from the Defects4J benchmark. The experiment results show that Neural-MBFL can localize more faults than traditional MBFL in terms of TOP-N (i.e., 9 for TOP-I, 17 for TOP-3 and 18 for TOP-5 on average) and MAP (i.e., 2.32% relative improvement on average). We also analyze the unique faults localized by Neural-MBFL and traditional MBFL. The statistical results show their complementarity. It motivates further analysis into the repair pattern distributions between Neural-MBFL and traditional MBFL to better understand their complementarity. By further comprehensive analysis of the repair pattern distribution, traditional MBFL has advantages in localizing faults related to rule-based code modifications. In contrast, Neural-MBFL has advantages in localizing complex faults requiring deep code comprehension. These findings show that incorporating neural mutation is promising in improving the effectiveness of MBFL.

Integrating Large Language Models (LLMs) into software fault localization represents a significant advancement in improving debugging efficiency for programmers. However, novice program fault localization, which is essential for computer science education, has not been thoroughly investigated in previous studies. In contrast to industrial programs target practical functionality, novice programs primarily deal with individual algorithmic issues. The distinct logic structures between novice and industrial programs can impact how effectively LLM understand and process them. Moreover, this difference reveals the inapplicability of the Competent Programmer Hypothesis, a fundamental assumption in industrial fault localization, to novice program fault localization. Therefore, industrial methodologies are unsuitable for novice programming, emphasizing the need for our empirical studies. To fill this gap, we evaluate LLMs’ effectiveness in localizing faults for novice programs in statement level. Using the widely used novice programs dataset Codeflaws and Condefects, we compare the performance of two commercial LLMs (i.e., ChatGPT-3.5 and ChatGPT-4) and three open-source LLMs (i.e., ChatGLM3, Llama2, and Code Llama) against traditional fault localization methods, examining their accuracy and overlap. Additionally, we investigate how prompt engineering improves localization precision. Our findings show ChatGPT-4’s overall superior performance, with ChatGPT-3.5 exhibiting minor advantages in certain cases. ChatGPT-4 outperforms the traditional methods with best performance by 592% and 137% on Codeflaws and Condefects. Specifically, each method exhibits unique strengths in localizing novice programming faults. Moreover, carefully crafted prompts can improve LLMs’ precision. These insights underscore the promising potential of utilizing LLMs for fault localization in novice programming.

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is [Formula: see text], [Formula: see text], [Formula: see text] and [Formula: see text] in efficiency, test suite size, diversity and robustness.

Summary Online judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐orientedMulti‐Objective Optimization‐BasedFault Localization‐OrientedTestCaseGeneration (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.

Abstract Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

Fault Localization (FL) is one of the most essential and time-consuming steps during software debugging. Mutation-based fault localization (MBFL) is one FL technique that has demonstrated promising fault localization accuracy in recent years. Current MBFL techniques mainly use First-Order Mutant (FOM) to localize faults, and only perform well in simple fault localization. When facing complex fault localization, MBFL with FOMs can only achieve low FL accuracy. Moreover, previous Higher-Order Mutant (HOM) generation techniques only use simple combinations of FOMs but do not consider the correlation between simple faults in the composition of complex faults. In this study, we consider the relationships between single faults and propose SFClu, a novel HOM generation method. Specifically, SFClu aims to generate HOMs to simulate complex faults consisting of multiple unrelated simple faults on multiple lines. To evaluate the performance of our proposed methods, we conduct empirical studies on 237 complex-fault programs from two datasets. The experimental results show that SFClu significantly outperforms traditional HOM generation methods (i.e., Last2First, DifferentOperators, and RandomMix). Furthermore, the experimental results also demonstrate that Higher-Order MBFL(HMBFL) with SFClu can outperform the state-of-the-art SBFL and MBFL techniques in terms of EXAM, TOP-N, and MAP metrics.

MBFL (Mutation-Based Fault Localization) is one of the most commonly studied fault localization techniques due to its promising fault localization effectiveness. However, MBFL incurs a high execution cost as it needs to execute the test suite on a large number of mutants. While previous studies have proposed mutant reduction methods for FOMs (First-Order Mutants) to help alleviate the cost of MBFL, the reduction of HOMs (Higher-Order Mutants) has not been thoroughly investigated. In this study, we propose SGS (Statement Granularity Sampling), a method which conducts HOMs reduction for HMBFL (Higher-Order Mutation-Based Fault Localization). Considering the relationship between HOMs and statements, we sample HOMs at the statement level to ensure each statement has corresponding HOMs. We empirically evaluate the fault localization effectiveness of HMBFL using SGS on 237 multiple-fault programs taken from the SIR and Codeflaws benchmarks. The experimental results show that (1) The best sampling ratio for HMBFL with SGS is 20%, which preserves the performance and reduces execution costs by 80% ; (2) The fault localization accuracy of HMBFL with SGS outperforms the state-of-the-art SBFL (Spectrum-Based Fault Localization) and MBFL techniques by 20%.

Spectrum-Based Fault Localization (SBFL) technique is widely applied for fault localization, identifying faulty statements potentially resulting in unexpected faulty programs’ behavior. However, researchers have approved that Coincidental Correct (CC) test cases contained in test suites can negatively affect the accuracy of SBFL. Previous researchers sought to identify CC test cases through machine learning algorithms, but the feature representation is insufficient, leading to limited accuracy. To address this challenge, we propose the Machine Learning-based CC test cases Identification approach (MLCCI), which leverages multiple features extracted from the program under test to identify CC test cases and map the CC identification task to a learning problem. To evaluate the performance of MLCCI, we conduct experiments in the well-known dataset Defects4J. The experimental results compared with state-of-the-art baselines indicate that: (1) MLCCI achieves higher CC identifying accuracy, with the average Recall, P recision, and F -measure values of MLCCI are 65.93%, 71.69%, and 53.74%, respectively; (2) The fault localization accuracy of MLCCI with the Jaccard formula outperforms baselines, where the values of Accuracy@ 1, 3, and 5 are 347, 369, and 393, achieving the maximum 137.67%, 67.73%, and 47.74% improvement against baselines, respectively. Besides, we perform ablation analysis to reveal the effectiveness of features utilized in this study.

Mutation-based fault localization has shown promising accuracy in localizing faults due to its finer analysis granularity. However, the effectiveness is limited when dealing with diverse real-world systems and actual faults, which results from its inflexible suspiciousness calculation and oversimplification of information. In this work, we propose a novel Mutation-Based Fault Localization technique, GMBFL, which utilizes Graph representation to achieve multi-information cooperation to improve fault localization. GMBFL comprises two key components: a fine-grained graph-based representation to fully utilize the information of the program, and an effective suspiciousness measure using the graph neural network to learn useful features from the graph. We evaluate GMBFL on 243 real faulty programs from Defects4J. The experimental results show that GMBFL can surpass both the state-of-the-art learning-based fault localization technique and 70 commonly used SBFL and MBFL techniques. In particular, GMBFL localizes 125 faults within TOP-1 whereas the best baseline technique can at most localize 109 faults within TOP-1.

The development of correct and efficient software can be hindered by compilation errors, which must be fixed to ensure the code's syntactic correctness and program language constraints. Neural network-based approaches have been used to tackle this problem, but they lack guarantees of output correctness and can require an unlimited number of modifications. Fixing compilation errors within a given number of modifications is a challenging task. We demonstrate that finding the minimum number of modifications to fix a compilation error is NP-hard. To address compilation error fixing problem, we propose OrdinalFix, a complete algorithm based on shortest-path CFL (context-free language) reachability with attribute checking that is guaranteed to output a program with the minimum number of modifications required. Specifically, OrdinalFix searches possible fixes from the smallest to the largest number of modifications. By incorporating merged attribute checking to enhance efficiency, the time complexity of OrdinalFix is acceptable for application. We evaluate OrdinalFix on two datasets and demonstrate its ability to fix compilation errors within reasonable time limit. Comparing with existing approaches, OrdinalFix achieves a success rate of 83.5 %, surpassing all existing approaches (71.7%).

Code comments are essential for software development and maintenance, as they provide natural language descriptions of the code that help developers understand the program and reduce the time spent on comprehension. However, writing code comments can be tedious and time-consuming, and many software projects lack comprehensive and up-to-date comments, which can impair the readability and maintainability of programs.

Automatically localizing faulty statements is a desired feature for effective learning programming. Most of the existing automated fault localization techniques are developed and evaluated on commercial or well-known open-source projects, which performed poorly on novice programs. In this paper, we propose a novel fault localization technique VsusFL (Variable-suspiciousness-based Fault Localization) for novice programs. VsusFL is inspired by simulating the manual program debugging process and takes advantage of variable value sequences. VsusFL can trace variable value changes, determine whether the intermediate state of the variables is correct, and report the potential faulty statements for novice programs. This paper presents the implementation of VsusFL and conducts empirical studies on 422 real faulty novice programs. Experimental results show that VsusFL performs much better than Grace, ANGELINA, VSBFL, Spectrum-Based Fault Localization (SBFL), and Variable-based Fault Localization (VFL) in terms of T O P -1, T O P -3, and T O P -5 metrics. Specifically, VsusFL can localize 90%, 35% and 9% more faulty statements than the best-performing baseline Grace. Moreover, We analyze the correlation between VsusFL and other techniques and find a weak correlation since they perform well on different programs, indicating the potential to further enhance fault localization performance through strategic integration of VsusFL with other methods.

Abstract Compilation errors are unavoidable during the debugging process of novice students. Compiler error messages can help novices to localize and remove errors, but these messages are difficult to understand for students. Previous studies have investigated the compilation error categorization by analyzing compiler error messages, but the categorization cannot cover all kinds of errors, which limits the evaluation of compilation error studies. Therefore, a comprehensive categorization for compilation errors is needed for evaluating the performance of models or tools related to compilation errors. In this study, we first propose a new compilation error categorization, which is based on the smallest unit of the program, tokens. The experiments on 29,573 programs from three datasets show that our proposed compilation error categorization can cover more types of errors and the distribution of the error categorization are significantly different between the datasets. Then, based on our proposed categorization, we develop a neural network model CLACER (CLAssification of Compilation ERrors) for predicting the compilation errors. The results indicate that CLACER can improve the compiler's error localization accuracy and predicts the compilation error effectively. Moreover, based on the proposed categorization, we conduct empirical studies to evaluate the performance of three repairing tools (i.e., DeepFix, RLAssist, and MACER). The comparison results illustrate that DeepFix and RLAssist can fix more errors in the category of delimiter than errors in other categories. Furthermore, MACER performs better than DeepFix and RLAssist because it has a sufficient repairing pattern set for the errors. We also provide some suggestions for improving the repairing tools in the future.

Abstract Identifying the location of faults in real‐world programs is one of the costly processes during software debugging. To reduce the debugging effort, various fault localization techniques have been proposed in recent years. Spectrum‐based fault localization (SBFL) is one kind of widely investigated fault localization technique. Most SBFL techniques first calculate the suspiciousness of program elements (such as statements, methods) to be faulty using the coverage information and execution results of tests. Then a rank list of program elements is generated according to their suspiciousness. However, some SBFL techniques only consider the binary coverage information (i.e., whether the program element is covered) but ignore some of the tests' running behaviors, such as the execution frequency when faults occur in the iteration entities or loop bodies, which are more likely to be faulty followed the propagation‐infection‐execution model. The execution frequency based techniques only replace the feature items of the existing formula limiting their effectiveness in fault localization. In this article, we propose a fault localization technique, class reduction and method call frequency (CRMF), which utilizes mutation analysis and information retrieval techniques. In particular, CRMF first uses mutation analysis to identify and reduce the classes, in which the program elements with a low probability of being faulty. Then we propose a new suspiciousness formula that applies information retrieval and considers method call frequency. To evaluate the effectiveness of CRMF, we conduct empirical studies on 264 real‐world programs from the Defects4J benchmark. Final results show that CRMF outperforms the statement frequency based technique FLSF and SBFL techniques (i.e., Ochiai, OP2, Tarantula, and Dstar) in both single‐fault programs and multiple‐fault programs. Specifically, CRMF can rank 29, 74, and 112 faults at the top 1, 3, 5 ranks and achieve a higher mean reciprocal rank for single‐fault programs and multiple‐fault programs. Finally, we discuss the essence of CRMF and analyze its effectiveness on multi‐fault programs in detail.

Automated code comment generation technologies can help developers understand code intent, which can significantly reduce the cost of software maintenance and revision. The latest studies in this field mainly depend on deep neural networks, such as convolutional neural networks and recurrent neural network. However, these methods may not generate high-quality and readable code comments due to the long-term dependence problem, which means that the code blocks used to summarize information are far from each other. Owing to the long-term dependence problem, these methods forget the previous input data’s feature information during the training process. In this article, to solve the long-term dependence problem and extract both the text and structure information from the program code, we propose a novel improved-Transformer-based comment generation method, named SeTransformer. Specifically, the SeTransformer utilizes the code tokens and an abstract syntax tree (AST) of programs to extract information as the inputs, and then, it leverages the self-attention mechanism to analyze the text and structural features of code simultaneously. Experimental results based on public corpus gathered from large-scale open-source projects show that our method can significantly outperform five state-of-the-art baselines (such as Hybrid-DeepCom and AST-attendgru). Furthermore, we also conduct a questionnaire survey for developers, and the results show that the SeTransformer can generate higher quality comments than those of other baselines.

Mutation-Based Fault Localization (MBFL) is one of the most widely studied techniques. MBFL adopts mutation analysis to generate mutants for revealing potential faults in the program. Previous studies proposed approaches to optimize MBFL in terms of efficiency and accuracy. However, these approaches ignored the difference of mutants on correct entities (such as statements) and faulty entities, which we refer this kind of difference as mutant bias. In this study, we identify and analyze the impact of mutant bias on MBFL. We find that the mutant bias may introduce effects to statement suspiciousness and negatively influence the fault localization accuracy of MBFL. To mitigate the mutant bias, we propose Delta4Ms, a model that captures the mutant bias from the mutants of the same statements. Then the real suspiciousness is obtained by removing the bias from the practical suspiciousness. To evaluate the performance of our proposed method, we conduct experimental studies on 320 real-world programs from Codeflaws. The experimental results show that Delta4Ms improves the fault localization accuracy of MBFL. Besides, Delta4Ms outperforms the state-of-the-art SBFL and three MBFL techniques significantly. Moreover, Delta4Ms ranks 94 and 161 of the target faults within the top-5 suspicious statements in single-fault and multiple-fault programs, respectively.

Fault localization is one of the most expensive activities in software debugging. Mutation-based fault localization (MBFL) is a commonly studied technique that applied mutation analysis to find the location of faults in the programs. Previous studies showed that MBFL adopted First-Order-Mutants (FOMs) that could achieve promising results in single-fault localization, but it did not perform well in multiple-fault localization. Recently, Higher-Order-Mutants (HOMs) were proposed for modeling complex faults but whether HOMs can help in fault localization is still unknown. In this paper, we investigate the performance of MBFL with FOMs and HOMs on single- and multiple-fault localization. Moreover, to study the characteristics of HOMs, we divide HOMs into three groups (i.e. Accurate HOMs, Partially accurate HOMs, and Inaccurate HOMs) by considering different mutation locations. Based on the empirical results on 186 versions of six real-world programs, we find that (1) In single-fault localization, FOMs can achieve better performance than HOMs. (2) However, in multiple-fault localization, HOMs (2-HOMs) localize more faults than FOMs. (3) Furthermore, different types of HOMs have different fault localization effectiveness, where Accurate HOMs outperform the other two HOMs categories. Therefore, the researchers should propose methods to find HOMs more useful for fault localization.

First-order mutants (FOMs) have been widely used in mutation-based fault localization (MBFL) approaches and have achieved promising results in single-fault localization scenarios (SFL-scenario). Higher-order mutants (HOMs) are proposed to simulate complex faults and can be applied in MBFL theoretically for multiple-fault localization scenarios (MFL-scenario). However, whether HOMs can improve MBFL’s performance is not investigated and the effectiveness is not thoroughly evaluated. In this empirical study, we investigate the impact of HOMs on the performance of MBFL in SFL-scenario and MFL-scenario. The experiments on two real-world benchmarks reveal that 1) 2-HOMs can help improve the MBFL performance in SFL-scenarios; 2) in MFL-scenarios, both 2-HOMs and 3-HOMs can achieve better performance than FOMs; and 3) huge computational cost cannot be ignored in the practice of HOMs. Therefore, effective methods to reduce the number of HOMs for future MBFL studies should be considered.

To improve the efficiency of the fault localization process, different automatic fault localization approaches have been proposed. Among these approaches, the spectrum-based fault localization (SBFL) approach has been widely used and studied due to its lightweight and high effectiveness. However, while the existence of coincidental correct (CC) test cases can influence the usefulness of SBFL in single-fault programs, their influence on multiple fault programs has not been thoroughly investigated. Therefore, in this article, we conduct a theoretical analysis and an empirical study to investigate the effect of CC test cases on multiple fault localization. The theoretical analysis is based on a suspiciousness calculation formula of SBFL, which divides CC test cases into three categories (specific, irrelevant, and unspecific) according to their association with a specific faulty statement. Following this analysis, we conduct an empirical study on two well-known open-source repositories (SIR and Defects4J), and the experimental results verify the correctness of our theoretical analysis. Specifically, reducing the number of specific CC test cases for a faulty statement can improve or maintain fault localization accuracy, while eliminating irrelevant CC test cases can have a negative effect. Finally, we design a CC test case identification solution based on the isolation-based multiple fault localization approach and demonstrate its effectiveness via a simulation experiment.

Compilation errors happen during the debugging process of novice students. Compiler error messages help novices to localize and remove errors, but these messages are difficult to understand for students. Some computing education researchers analyzed the compiler error messages generated by novice’s attempts to compile their programs. However, some important questions remain open. For example, the existing compilation error category cannot cover all programs due to the simple static analysis and program repair patterns. And existing prediction models for classifying compilation errors are unsatisfactory because of the inappropriate neural networks. In this paper, we first propose a new category of compilation error based on the program tokens, which is the smallest unit of the program. Then we develop a neural network model CLACER (ClAssification of Compilation ERrors) based on TextCNN. CLACER performs better on extracting semantic features and statistical features from compiler error messages. To verify the effectiveness of our proposed category and corresponding method CLACER, we choose 16,926 student programs as our experimental subjects. Final experimental results indicate that our proposed classification category covers 16.5% more programs than the state-of-the-art category TEGCER. Moreover, CLACER improves the compiler’s localization effectiveness and with a 4.25% improvement on the TEGCER category. Further analysis shows that CLACER has a promising prediction performance for different error classes, and TextCNN is more suitable for constructing the compilation error classification model.

Test Case Prioritization (TCP) based on the continuous decision of Reinforcement Learning (RL) has achieved a successful application for test cases optimization in Continuous Integration (CI). The reward functions of RL describe how a test case "ought" to be executed in next integration, of which the design is usually based on the historical executions of the test case. The Average Percentage of Historical Failure (APHF) had been considered as one of the best reward function which has a strong correlation with the recent failure executions of a test case. However, for a test case with many historical failures but passes in recent cycles, the APHF value may be low. In this paper, two novel reward functions are proposed focusing on the impact of failure position in test case history execution sequence, which are the Average Position Exponential Weight (APEW) reward function and the Average Position Quadratic Weight (APQW) reward function, respectively. Both APEW and APQW carry out weight design of failure position but with different weights. We theoretically prove the issue of the only strong correlation with recent failure executions, and also prove that both proposed reward functions can reflect the quantity of historical failures and the distribution of these failures. Experimental verification on 10 industrial-level data sets show that the proposed reward functions can effectively improve the fault detection capability of test cases.

Identifying the location of faults in real-world programs is one of the most costly processes during software debugging. In order to reduce debugging effort, many fault localization techniques have been proposed. One of the most widely studied technique is called Spectrum-based fault localization (SBFL), which uses the coverage information and execution results of test cases to do fault localization. Most SBFL techniques only consider the binary coverage information and ignore the execution frequency, so their fault localization accuracy is limited, especially when faults occur in the iteration entities or loop bodies. In this paper, we propose IRBFL, a novel fault localization technique based on information retrieval to extract information from execution frequencies of program entities. IRBFL uses mutation analysis to reduce the low suspicious classes, and then it adopts information retrieval techniques to calculate the suspiciousness value. We evaluate IRBFL on 205 real-world faults from 5 programs in Defects4J benchmark. The experimental results show that our proposed method outperforms the other five state-of-the-art SBFL techniques. More specifically, no matter in single-fault or multi-fault programs, IRBFL can identify 2 to 3 times more faulty methods than the other five SBFL techniques when checking the top 1 method. More empirical results in terms of other metrics, including acc@3, acc@5, EXAM, MRR, and MAP, also indicate that IRBFL technique is better than the other five SBFL techniques.

Online Judge (OJ) system, which can automatically evaluate the results (right or wrong) of programs by executing them on standard test cases, is widely used in programming education. While an OJ system with personalized feedback can not only give execution results, but also provide information to assist students in locating their problems quickly. Automatically fault localization techniques are designed to find the exact faults in programs automatically, experimental results showed their effect on locating artificial faults, but their effectiveness on novice programs needs to be investigated. In this paper, we first evaluate the effectiveness of several widely-studied fault localization techniques on novice programs, and then we use fine-grained test cases to improve the fault localization accuracy. Empirical studies are conducted on 77 real student programs and the results show that, compared with original test cases in OJ system, the fault localization accuracy can be improved obviously when using fine-grained test cases. More specifically, in terms of TOP-1, TOP-3 and TOP-5 metrics, the maximum results can be improved from 5, 22, 37 to 9, 24, 48, respectively. The results indicate that more faults can be located when checking the top 1, 3 or 5 statements, so the fault localization accuracy is enhanced. Furthermore, a Test Case Granularity (TCG) concept is introduced to describe fine-grained test cases, and empirically studies demonstrate that there is a strong correlation between TCG and fault localization accuracy.

Bug isolation is the main approach to multi-fault localization, where failed test cases are divided into groups, and each group failed test cases are used to localize a single fault combined with all passed test cases. Ideally, all failed test cases within a single group execute the same faulty statements. However, misgrouping usually occurs due to the clustering algorithms may not able to divide failed test cases accurately. This paper focuses on the impact of fault localization by the accuracy of the clustering algorithm. A large quantitative empirical study is conducted on 12786 version programs with multiple faults, in which the misgrouping are simulated with different accuracy by a controlled experiment. The results indicate that the effect of fault localization will become worse as the accuracy of clustering decreases.

The commonly-used software fault localization approaches mainly utilize test coverage information and test cases execution results to calculate the suspiciousness of each program entity to identify the location of faults, namely spectrum based software fault localization (SBFL). It had been argued that such techniques are not helpful in real debugging process, since the low accuracy of localization and few information provided to programmers. In this paper we consider the combination of statement based fault classification with the SBFL, aiming at increasing accuracy of fault localization and provide additional possible fault information to programmers. An improved technique, fault classification oriented SBFL (FC-SBFL), is proposed in this paper, in which the suspiciousness value is adjusted dynamically based on the probability of statement being faulty. Experimental results on real application programs show that FC-SBFL is more effective than SBFL to locate faults, and studies with Tarantula and OP2 show that more than 75% faults have been identified in a better effectiveness.

Mutation Based Fault Localization(MBFL) is a fault localization technique based on mutation analysis, which precisely identifies the location of fault but incurs a high execution cost, since it needs to execute the test suite on a large amount of mutants. Reduction strategies proposed are usually regarding selecting mutation operators or sampling mutants directly, meanwhile at the cost of losing precision of fault localization. This paper proposes a Statement-Oriented Mutant Reduction strategy (SOME), which selects a proportion of mutants at the statement level, specifically, the statements covered by failed tests. SOME keeps the advantage of using whole types of mutation operators, and further considers the increase of mutants' diversity to avoid the precision loss of fault localization. Empirical studies are conducted on 112 faulty versions from 7 benchmark programs, and the results indicate that SOME can reduce 73.51%-79.98% mutation execution cost while keeping almost the same fault location precision as the original MBFL without reduction..