The widespread use of Large Language Models (LLMs) in software engineering has intensified the need for improved model and resource efficiency. In particular, for neural code generation, LLMs are used to translate function/method signature and DocString to executable code. DocStrings, which capture user requirements for the code and are typically used as the prompt for LLMs, often contain redundant information. Recent advancements in prompt compression have shown promising results in Natural Language Processing (NLP), but their applicability to code generation remains uncertain. Our empirical study shows that the state-of-the-art prompt compression methods achieve only about 10% reduction, as further reductions would cause significant performance degradation. In our study, we propose a novel compression method, ShortenDoc, dedicated to DocString compression for code generation. Our experiments on six code generation datasets, five open source LLMs (1B to 10B parameters), and one closed-source LLM GPT-4o confirm that ShortenDoc achieves 25–40% compression while preserving the quality of generated code, outperforming other baseline methods at similar compression levels. The benefit of this method is to improve efficiency and reduce the token processing cost while maintaining the quality of the generated code, especially when calling third-party APIs.

Code Language Models (CLMs), particularly those leveraging deep learning, have achieved significant success in code intelligence domain. However, the issue of security, particularly backdoor attacks, is often overlooked in this process. The previous research has focused on designing backdoor attacks for CLMs, but effective defenses have not been adequately addressed. In particular, existing defense methods from natural language processing, when directly applied to CLMs, are not effective enough and lack generality, working well in some models and scenarios but failing in others, thus fall short in consistently mitigating backdoor attacks. To bridge this gap, we first confirm the phenomenon of “early learning” as a general occurrence during the training of CLMs. This phenomenon refers to that a model initially focuses on the main features of training data but may become more sensitive to backdoor triggers over time, leading to overfitting and susceptibility to backdoor attacks. We then analyze that overfitting to backdoor triggers results from the use of the cross-entropy loss function, where the unboundedness of cross-entropy leads the model to increasingly concentrate on the features of the poisoned data. Based on this insight, we propose a general and effective loss function DeCE (Deceptive Cross-Entropy) by blending deceptive distributions and applying label smoothing to limit the gradient to bounded, which prevents the model from overfitting to backdoor triggers and then enhances the security of CLMs against backdoor attacks. To evaluate the effectiveness of our defense method, we select four code-related tasks as our experimental scenes and conduct experimental analyses on both natural language and two programming languages (Java and Python). Our experiments across multiple models with different sizes (from 125 millions to 7 billions) and poisoning ratios demonstrate the applicability and effectiveness of DeCE in enhancing the security of CLMs. The findings emphasize the potential of DeCE as a novel defense mechanism for CLMs, effectively tackling the challenge of securing models against backdoor threats.

Deep Learning (DL) frameworks are fundamental components of DL systems in their development, deployment, and execution, while defects in DL frameworks can cause severe consequences. Ensuring the quality of DL frameworks has therefore become a pressing challenge. Among the various testing techniques, model mutation has emerged as a widely adopted approach. Such methods generate mutants by applying mutation operators to DL models (e.g., structural changes or parameter edits) and then analyzing inconsistencies, crashes, or abnormal behaviors across different frameworks or hardware. Despite its effectiveness, existing methods suffer from the following limitations. First, they mainly reuse operators designed for model testing, raising doubts about their ability to expose framework-level defects. Besides, they insufficiently consider mutation constraints, such as mutation type, position, and order, which directly affect the defect detection ability of generated mutants. Finally, they rely on the limited detection range and narrow test oracles, focusing on functional correctness in model inference while overlooking defects in efficiency, resource usage, and other defects that developers care about in other stages, such as model training or deployment. These limitations result in a weak alignment with the critical defects that developers are most concerned about in practice. Motivated by these observations, this study conducts a comprehensive investigation into the effectiveness of existing mutation-based testing methods. We first collect and classify defect reports from PyTorch and MindSpore according to developers’ priority tags, building a taxonomy of seven categories and 19 sub-categories of HP defects. We then map the defects reported by five state-of-the-art methods into this taxonomy to evaluate their detection abilities. To explain these limitations, we further analyze how three key factors, mutation type, mutation position, and mutation order, affect the generated mutants. Based on the experiment results, we summarize ten findings ranging from revealing the priority of developers on fixing framework defects, evaluating the defect detection ability of existing methods, to how mutation factors affect the generated mutants. Furthermore, we reveal four limitations and their root causes of existing methods and propose four targeted optimization strategies. We further apply these strategies to COMET and successfully uncover six new defects spanning four types, including two previously unreported categories. Overall, our study identifies 38 unique framework defects, of which 30 are confirmed by developers and 12 have been fixed, demonstrating the practical value of our findings.

Smart contracts have become a foundational component of blockchain systems, enabling decentralized, transparent, and autonomous execution of application logic across various domains, including decentralized finance (DeFi), gaming, and digital identity. Due to their immutable and trustless nature, smart contracts often manage and transfer substantial amounts of assets without human intervention. However, vulnerabilities in smart contracts can lead to substantial financial losses. Among these, access control vulnerabilities are particularly critical, typically originating from inadequately designed or incorrectly implemented permission mechanisms. Most existing methods for detecting access control vulnerabilities are based on static analysis, which heavily relies on manually defined rules and pattern matching. While these methods are efficient at identifying certain classes of known vulnerabilities, they are inherently limited in scope and generalization. In particular, they often fail to capture the underlying business logic of smart contracts.In this paper, we propose an LLM-based multi-agent system, named ACTaint, for detecting access control vulnerabilities in Solidity smart contracts. ACTaint first performs static analysis to guide the sink agent in identifying potential sinks. Then, based on the identified sinks, the taint agent conducts taint analysis to determine whether a data flow exists from untrusted sources to these sinks. We evaluate our approach on three datasets: known CVE cases, a set of 624 real-world smart contracts, and another set of 93 real-world smart contracts. The results demonstrate that our method outperforms existing tools in both datasets. On the first dataset, our approach outperforms state-of-the-art tools, including AChecker and GPTLens, achieving higher recall and F1-score. On the second dataset, our method surpasses the leading static analysis tool AChecker, with a 8.3% improvement in precision and an 9.7% improvement in F1-score.

Modern computing architectures (e.g., multi-core CPUs, GPUs, distributed systems) rely on parallel code implemented via frameworks such as OpenMP, MPI, and CUDA. While large language models (LLMs) have shown strong performance in general code generation, they struggle with the structured reasoning required for parallel programming, such as handling concurrency, synchronization, and framework-specific semantics. In practical parallel code development, a common workflow begins with sequential code and incrementally introduces parallel directive codes. We formalize this process as the task of framework-based parallel code completion (FPCC), which involves three subtasks: identifying insertion points, selecting parallel frameworks, and completing parallel directive codes.To support this task, we construct a high-quality dataset of 16,638 framework-based parallel code pairs across six widely used frameworks, labeled with directive points, parallel frameworks, and the code of parallel directives. However, our empirical results show that six popular LLMs perform poorly on FPCC, particularly struggling with identifying insertion points and completing correct directive codes.To address these limitations, we propose HPCL, a curriculum-based fine-tuning framework that progressively improves model capabilities in insertion point identification, parallel framework selection, and parallel directive code completion. Our approach achieves substantial improvements, yielding an 17.82% increase in EM and a 5.43% improvement in DIR scores over LLM-based baselines. Finally, expert-guided error analysis reveals common failure patterns and suggests future directions, such as in retrieval-augmented completion and consistency-aware training.

Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose Code-DiTing, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1-671B to our Code-DiTing 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, Code-DiTing 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. Code-DiTing 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that Code-DiTing is robust to preference leakage and can serve as a promising alternative for code evaluation.

Large Language Models (LLMs) can pose serious risks in real-world applications due to their potential for erroneous behavior, necessitating comprehensive and effective testing of LLMs. To assess the robustness of LLMs, adversarial attacks are typically conducted by constructing adversarial examples. Previous methods often require extensive queries and access to the internal information of the victim model. However, the internal information of most black-box LLMs is not accessible, rendering these testing methods infeasible. In addition, excessive queries to commercial black-box LLMs may incur substantial costs. To address these issues, this paper proposes an E nsemble-based T ransfer T esting method for L arge L anguage M odels (ETTLLM). In contrast to previous adversarial testing methods for LLMs, ETTLLM queries white-box surrogates rather than the victim model, thereby significantly reducing testing costs. Moreover, it enhances the transferability and generalization of adversarial examples across diverse real-world classification tasks. Compared to baselines, ETTLLM significantly reduced the number of queries to the victim model, with an average of 1.6 queries, just 1.2% of the baselines. Furthermore, the textual similarity and modification rate of the adversarial examples generated by ETTLLM differ from the baselines by no more than 1.6%, while achieving 70% of the attack success rate compared to the baselines.

Several code comment generation approaches based on sequence-to-sequence (Seq2Seq) models have been proposed. Such approaches often extract structure information from abstract syntax trees (ASTs) using a certain serialization method. However, some structural information is inevitably lost while serializing ASTs. Furthermore, existing serialization methods only consider the “type” attribute of the nodes, neglecting the “value” attribute of the nodes. To further improve the performance of code comment generation, we propose a code comment generation approach, called SCATCom, which integrates a more comprehensive set of information from source code and ASTs, encompassing semantic, sequential, syntactic, and hierarchical structure information for code comment generation. Meanwhile, an AST traversal method, called V-POT, is presented, which considers both the “type” and the “value” attributes of the nodes. Experiments were designed and conducted on two commonly used datasets to validate the performance of our approach and the impact of five different serialization ways of ASTs on two code comment generation methods. The BLEU, METEOR, and ROUGE scores for our approach reach 52.6, 34.16, and 63.26 with an improvement of [Formula: see text], [Formula: see text], and [Formula: see text] compared to the baselines. It is evident that V-POT, which retains both the “type” and the “value” attributes, is superior to other methods that use only the “type” attribute.

Context: Software vulnerability assessment (SVA) is critical for identifying, evaluating, and prioritizing security weaknesses in software applications. Objective: Despite the increasing application of large language models (LLMs) in various software engineering tasks, their effectiveness in SVA remains underexplored. Method: To address this gap, we introduce a novel approach SVA-ICL, which leverages in-context learning (ICL) to enhance LLM performance. Our approach involves the selection of high-quality demonstrations for ICL through information fusion, incorporating both source code and vulnerability descriptions. For source code, we consider semantic, lexical, and syntactic similarities, while for vulnerability descriptions, we focus on textual similarity. Based on the selected demonstrations, we construct context prompts and consider DeepSeek-V2 as the LLM for SVA-ICL. Results: We evaluate the effectiveness of SVA-ICL using a large-scale dataset comprising 12,071 C/C++ vulnerabilities. Experimental results demonstrate that SVA-ICL outperforms state-of-the-art SVA baselines in terms of Accuracy, F1-score, and MCC measures. Furthermore, ablation studies highlight the significance of component customization in SVA-ICL, such as the number of demonstrations, the demonstration ordering strategy, and the optimal fusion ratio of different modalities. Conclusion: Our findings suggest that leveraging ICL with information fusion can effectively improve the effectiveness of LLM-based SVA, warranting further research in this direction.

Abstract As software grows in size and complexity, software vulnerabilities are increasing, leading to a range of serious insecurity issues. Open‐source software vulnerability reports and documentation can provide researchers with great convenience for analysis and detection. However, the quality of different data sources varies, the data are duplicated and lack of correlation, which often requires a lot of manual management and analysis. In order to solve the problems of scattered and heterogeneous data and lack of correlation in traditional vulnerability repositories, this paper proposes a software vulnerability feature knowledge extraction method that combines the N‐gram model and mask similarity. The method generates mask text data based on the extraction of N‐gram candidate keywords and extracts vulnerability feature knowledge by calculating the similarity of mask text. This method analyzes the samples efficiently and stably in the environment of large sample size and complex samples and can obtain high‐value semi‐structured data. Then, the final node, relationship, and attribute information are obtained by secondary knowledge cleaning and extraction of the extracted semi‐structured data results. And based on the extraction results, the corresponding software vulnerability domain knowledge graph is constructed to deeply explore the semantic information features and entity relationships of vulnerabilities, which can help to efficiently study software security problems and solve vulnerability problems. The effectiveness and superiority of the proposed method is verified by comparing it with several traditional keyword extraction algorithms on Common Weakness Enumeration (CWE) and Common Vulnerabilities and Exposures (CVE) vulnerability data.

In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).

In recent years, pre-trained language models have seen significant success in natural language processing and have been increasingly applied to code-related tasks. Code intelligence tasks have shown promising performance with the support of code pre-trained language models. Pre-processing code simplification methods have been introduced to prune code tokens from the model’s input while maintaining task effectiveness. These methods improve the efficiency of code intelligence tasks while reducing computational costs. Post-prediction code simplification methods provide explanations for code intelligence task outcomes, enhancing the reliability and interpretability of model predictions. However, comprehensive evaluations of these methods across diverse code pre-trained model architectures and code intelligence tasks are lacking. To assess the effectiveness of code simplification methods, we conduct an empirical study integrating these code simplification methods with various pre-trained code models across multiple code intelligence tasks. Our empirical findings suggest that developing task-specific code simplification methods would be beneficial. Then, we recommend leveraging post-prediction methods to summarize prior knowledge, which can pre-process code simplification strategies. Moreover, establishing more evaluation mechanisms for code simplification is crucial. Finally, we propose incorporating code simplification methods into the pre-training phase of code pre-trained models to enhance their program comprehension and code representation capabilities.

Convolutional Neural Networks (CNNs) have been widely used in various fields. However, it is essential to perform sufficient testing to detect internal defects before deploying CNNs, especially in security-sensitive scenarios. Generating error-inducing inputs to trigger erroneous behavior is the primary way to detect CNN model defects. However, in practice, when the model under test is a black-box CNN model without accessible internal information, in some scenarios it is still necessary to generate high-quality test inputs within a limited testing budget. In such a new scenario, a potential approach is to generate transferable test inputs by analyzing the internal knowledge of other white-box CNN models similar to the model under test, and then use transferable test inputs to test the black-box CNN model. The main challenge in generating transferable test inputs is how to improve their error-inducing capability for different CNN models without changing the test oracle. We found that different CNN models make predictions based on features of similar important regions in images. Adding targeted perturbations to important regions will generate transferable test inputs with high realism. Therefore, we propose the Interpretable Analysis-based Transferable Test (IATT) Generation method for CNNs, which employs interpretation methods of CNN models to explain and localize important regions in test inputs, using backpropagation optimizer and perturbation mask process to add targeted perturbations to these important regions, thereby generating transferable test inputs. This process is repeated to iteratively optimize the transferability and realism of the test inputs. To verify the effectiveness of IATT, we perform experimental studies on nine deep learning models, including ResNet-50 and Vit-B/16, and commercial computer vision system Google Cloud Vision , and compared our method with four state-of-the-art baseline methods. Experimental results show that transferable test inputs generated by IATT can effectively cause black-box target models to output incorrect results. Compared to existing testing and adversarial attack methods, the average Error-inducing Success Rate (ESR) in different testing scenarios is 18.1%–52.7% greater than the baseline methods. Additionally, the test inputs generated by IATT achieve high ESR while maintaining high realism.

The Masked Language Modeling (MLM) task is widely recognized as one of the most effective pre-training tasks and currently derives many variants in the Software Engineering (SE) field. However, most of these variants mainly focus on code representation without distinguishing between different code token types, while some focus on a specific type, such as code identifiers. Indeed, various code token types exist, and there is no evidence that only identifiers can improve PTMs. Thus, to improve PTMs through different types, we conducted an extensive study to evaluate how different type-specific masking tasks can affect PTMs. First, we extract five code token types, convert them into type-specific masking tasks, and generate their combinations. Second, we pre-train CodeBERT and PLBART using combinations and fine-tuned them on four SE downstream tasks. Experimental results show that type-specific masking tasks can enhance CodeBERT and PLBART on all downstream tasks. Furthermore, we discuss topics related to low-resource datasets, conflicting PTMs that original pre-training tasks conflict with our methods, the cost and performance of our methods, factors that impact the performance of our methods, and applying our methods on state-of-the-art PTMs. These discussions comprehensively analyze the strengths and weaknesses of different type-specific masking tasks.

Pre-trained models (PTMs) have succeeded in various software engineering (SE) tasks following the “pre-train then fine-tune” paradigm. As fully fine-tuning all parameters of PTMs can be computationally expensive, a potential solution is parameter-efficient fine-tuning (PEFT), which freezes PTMs while introducing extra parameters. Although PEFT methods have been applied to SE tasks, researchers often focus on specific scenarios and lack a comprehensive comparison of PTMs from different aspects such as field, size, and architecture. To fill this gap, we have conducted an empirical study on six PEFT methods, eight PTMs, and four SE tasks. The experimental results reveal several noteworthy findings. For example, model architecture has little impact on PTM performance when using PEFT methods. Additionally, we provide a comprehensive discussion of PEFT methods from three perspectives. First, we analyze the effectiveness and efficiency of PEFT methods. Second, we explore the impact of the scaling factor hyperparameter. Finally, we investigate the application of PEFT methods on the latest open source large language model, Llama 3.2. These findings provide valuable insights to guide future researchers in effectively applying PEFT methods to SE tasks.

As a key phase in software testing and debugging, fault localization can significantly influence the efficiency of fixing software faults. Among the various techniques, Mutation-Based Fault Localization (MBFL) is a widely studied fault localization technique that uses mutation analysis to guide the process of localizing faults. However, as the essential input source for MBFL, traditional mutation generates syntactical mutants, which cannot mimic the real faults and may affect the fault localization effectiveness. To address this issue, we resort to a code pre-trained model for program mutation, which is called neural mutation. Neural mutation can generate semantical mutants and even utilize the context information surrounding the mutation position. Based on the neural mutation, we propose Neural-MBFL by utilizing the high-quality mutants generated by neural mutation. To evaluate the effectiveness of Neural- MBFL, we conduct experiments on 393 faulty programs from the Defects4J benchmark. The experiment results show that Neural-MBFL can localize more faults than traditional MBFL in terms of TOP-N (i.e., 9 for TOP-I, 17 for TOP-3 and 18 for TOP-5 on average) and MAP (i.e., 2.32% relative improvement on average). We also analyze the unique faults localized by Neural-MBFL and traditional MBFL. The statistical results show their complementarity. It motivates further analysis into the repair pattern distributions between Neural-MBFL and traditional MBFL to better understand their complementarity. By further comprehensive analysis of the repair pattern distribution, traditional MBFL has advantages in localizing faults related to rule-based code modifications. In contrast, Neural-MBFL has advantages in localizing complex faults requiring deep code comprehension. These findings show that incorporating neural mutation is promising in improving the effectiveness of MBFL.

Deep learning (DL) frameworks are the fundamental infrastructure for various DL applications. Framework defects can profoundly cause disastrous accidents, thus requiring sufficient detection. In previous studies, researchers adopt DL models as test inputs combined with mutation to generate more diverse models. Though these studies demonstrate promising results, most detected defects are considered trivial (i.e., either treated as edge cases or ignored by the developers). To identify important bugs that matter to developers, we propose a novel DL framework testing method DevMuT, which generates models by adopting mutation operators and constraints derived from developer expertise. DevMuT simulates developers' common operations in development and detects more diverse defects within more stages of the DL model lifecycle (e.g., model training and inference). We evaluate the performance of DevMuT on three widely used DL frameworks (i.e., PyTorch, JAX, and Mind-Spore) with 29 DL models from nine types of industry tasks. The experiment results show that DevMuT outperforms state-of-the-art baselines: it can achieve at least 71.68% improvement on average in the diversity of generated models and 28.20% improvement on average in the legal rates of generated models. Moreover, DevMuT detects 117 defects, 63 of which are confirmed, 24 are fixed, and eight are of high value confirmed by developers. Finally, DevMuT has been deployed in the MindSpore community since December 2023. These demonstrate the effectiveness of DevMuT in detecting defects that are close to the real scenes and are of concern to developers.

Smart contracts are computer programs that run on a blockchain. As the functions implemented by smart contracts become increasingly complex, the number of cross-contract interactions within them also rises. Consequently, the combinatorial explosion of transaction sequences poses a significant challenge for smart contract security vulnerability detection. Existing static analysis-based methods for detecting cross-contract vulnerabilities suffer from high false-positive rates and cannot generate test cases, while fuzz testing-based methods exhibit low code coverage and may not accurately detect security vulnerabilities. The goal of this paper is to address the above limitations and efficiently detect cross-contract vulnerabilities. To achieve this goal, we present CrossFuzz, a fuzz testing-based method for detecting cross-contract vulnerabilities. First, CrossFuzz generates parameters of constructors by tracing data propagation paths. Then, it collects inter-contract data flow information. Finally, CrossFuzz optimizes mutation strategies for transaction sequences based on inter-contract data flow information to improve the performance of fuzz testing. We implemented CrossFuzz, which is an extension of ConFuzzius, and conducted experiments on a real-world dataset containing 396 smart contracts. The results show that CrossFuzz outperforms xFuzz, a fuzz testing-based tool optimized for cross-contract vulnerability detection, with a 10.58% increase in bytecode coverage. Furthermore, CrossFuzz detects 1.82 times more security vulnerabilities than ConFuzzius. Our method utilizes data flow information to optimize mutation strategies. It significantly improves the efficiency of fuzz testing for detecting cross-contract vulnerabilities.

Summary Online judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐orientedMulti‐Objective Optimization‐BasedFault Localization‐OrientedTestCaseGeneration (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.

Abstract Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

Pre-trained code generation models (PCGMs) have been widely applied in neural code generation, which can generate executable code from functional descriptions in natural languages, possibly together with signatures. Despite substantial performance improvement of PCGMs, the role of method names in neural code generation has not been thoroughly investigated. In this article, we study and demonstrate the potential of benefiting from method names to enhance the performance of PCGMs from a model robustness perspective. Specifically, we propose a novel approach, named neu RA l co D e gener A tor R obustifier (RADAR). RADAR consists of two components: RADAR -Attack and RADAR -Defense. The former attacks a PCGM by generating adversarial method names as part of the input, which are semantic and visual similar to the original input but may trick the PCGM to generate completely unrelated code snippets. As a countermeasure to such attacks, RADAR -Defense synthesizes a new method name from the functional description and supplies it to the PCGM. Evaluation results show that RADAR -Attack can reduce the CodeBLEU of generated code by 19.72% to 38.74% in three state-of-the-art PCGMs (i.e., CodeGPT, PLBART, and CodeT5) in the fine-tuning code generation task and reduce the Pass@1 of generated code by 32.28% to 44.42% in three state-of-the-art PCGMs (i.e., Replit, CodeGen, and CodeT5+) in the zero-shot code generation task. Moreover, RADAR -Defense is able to reinstate the performance of PCGMs with synthesized method names. These results highlight the importance of good method names in neural code generation and implicate the benefits of studying model robustness in software engineering.

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models ($\ell$LMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most$\ell$LMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approachCOTTONwhich can leverage$\ell$LMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated byCOTTONoutperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated byCOTTONboost various$\ell$LMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by Gemini and gpt-3.5-turbo. The results also reveal thatCOTTONnot only improves the performance of$\ell$LMs, but also enhances the performance of LLMs. Our study showcases the potential of$\ell$LMs in software engineering applications.

Recently, with the widespread use of deep neural networks (DNNs) in high-stakes decision-making systems (such as fraud detection and prison sentencing), concerns have arisen about the fairness of DNNs in terms of the potential negative impact they may have on individuals and society. Therefore, fairness testing has become an important research topic in DNN testing. At the same time, the neural network coverage criteria (such as criteria based on neuronal activation) is considered as an adequacy test for DNN white-box testing. It is implicitly assumed that improving the coverage can enhance the quality of test suites. Nevertheless, the correlation between DNN fairness (a test property) and coverage criteria (a test method) has not been adequately explored. To address this issue, we conducted a systematic empirical study on seven coverage criteria, six fairness metrics, three fairness testing techniques, and five bias mitigation methods on five DNN models and nine fairness datasets to assess the correlation between coverage criteria and DNN fairness. Our study achieved the following findings: 1) with the increase in the size of the test suite, some of the coverage and fairness metrics changed significantly, as the size of the test suite increased; 2) the statistical correlation between coverage criteria and DNN fairness is limited; and 3) after bias mitigation for improving the fairness of DNN, the change pattern in coverage criteria is different; 4) Models debiased by different bias mitigation methods have a lower correlation between coverage and fairness compared to the original models. Our findings cast doubt on the validity of coverage criteria concerning DNN fairness (i.e., increasing the coverage may even have a negative impact on the fairness of DNNs). Therefore, we warn DNN testers against blindly pursuing higher coverage of coverage criteria at the cost of test properties of DNNs (such as fairness).

In vulnerability detection approaches based on deep learning, fine-tuning with Pre-trained Language Models (PLMs) is a prevalent technique. Unfortunately, a natural gap exists between model pre-training tasks and vulnerability detection tasks due to different input formats, and the performance of fine-tuning relies on downstream dataset scales. Recently, prompt tuning has been used to alleviate these issues. However, it has not received enough attention in vulnerability detection. To assess the effectiveness of prompt tuning, we consider three classical vulnerability detection tasks: within-domain vulnerability detection, cross-domain vulnerability detection, and vulnerability type detection. Our empirical study considers three popular PLMs: CodeBERT, CodeT5, and CodeGPT. Then we use Devign, BigVul, and Reveal datasets as our experimental subjects. Our empirical results indicate that (1) compared to fine-tuning, prompt tuning can increase the accuracy of three tasks by an average of 42 %, 38%, and 41 %, respectively; (2) different prompt templates can have up to an 8 % impact on accuracy; (3) in data scarcity scenarios, the superiority of prompt tuning over fine-tuning is more obvious. Our research demonstrates that using prompt tuning can help to achieve better performance in vulnerability detection tasks and is a promising research direction in the future.

Fault Localization (FL) is one of the most essential and time-consuming steps during software debugging. Mutation-based fault localization (MBFL) is one FL technique that has demonstrated promising fault localization accuracy in recent years. Current MBFL techniques mainly use First-Order Mutant (FOM) to localize faults, and only perform well in simple fault localization. When facing complex fault localization, MBFL with FOMs can only achieve low FL accuracy. Moreover, previous Higher-Order Mutant (HOM) generation techniques only use simple combinations of FOMs but do not consider the correlation between simple faults in the composition of complex faults. In this study, we consider the relationships between single faults and propose SFClu, a novel HOM generation method. Specifically, SFClu aims to generate HOMs to simulate complex faults consisting of multiple unrelated simple faults on multiple lines. To evaluate the performance of our proposed methods, we conduct empirical studies on 237 complex-fault programs from two datasets. The experimental results show that SFClu significantly outperforms traditional HOM generation methods (i.e., Last2First, DifferentOperators, and RandomMix). Furthermore, the experimental results also demonstrate that Higher-Order MBFL(HMBFL) with SFClu can outperform the state-of-the-art SBFL and MBFL techniques in terms of EXAM, TOP-N, and MAP metrics.

MBFL (Mutation-Based Fault Localization) is one of the most commonly studied fault localization techniques due to its promising fault localization effectiveness. However, MBFL incurs a high execution cost as it needs to execute the test suite on a large number of mutants. While previous studies have proposed mutant reduction methods for FOMs (First-Order Mutants) to help alleviate the cost of MBFL, the reduction of HOMs (Higher-Order Mutants) has not been thoroughly investigated. In this study, we propose SGS (Statement Granularity Sampling), a method which conducts HOMs reduction for HMBFL (Higher-Order Mutation-Based Fault Localization). Considering the relationship between HOMs and statements, we sample HOMs at the statement level to ensure each statement has corresponding HOMs. We empirically evaluate the fault localization effectiveness of HMBFL using SGS on 237 multiple-fault programs taken from the SIR and Codeflaws benchmarks. The experimental results show that (1) The best sampling ratio for HMBFL with SGS is 20%, which preserves the performance and reduces execution costs by 80% ; (2) The fault localization accuracy of HMBFL with SGS outperforms the state-of-the-art SBFL (Spectrum-Based Fault Localization) and MBFL techniques by 20%.

Spectrum-Based Fault Localization (SBFL) technique is widely applied for fault localization, identifying faulty statements potentially resulting in unexpected faulty programs’ behavior. However, researchers have approved that Coincidental Correct (CC) test cases contained in test suites can negatively affect the accuracy of SBFL. Previous researchers sought to identify CC test cases through machine learning algorithms, but the feature representation is insufficient, leading to limited accuracy. To address this challenge, we propose the Machine Learning-based CC test cases Identification approach (MLCCI), which leverages multiple features extracted from the program under test to identify CC test cases and map the CC identification task to a learning problem. To evaluate the performance of MLCCI, we conduct experiments in the well-known dataset Defects4J. The experimental results compared with state-of-the-art baselines indicate that: (1) MLCCI achieves higher CC identifying accuracy, with the average Recall, P recision, and F -measure values of MLCCI are 65.93%, 71.69%, and 53.74%, respectively; (2) The fault localization accuracy of MLCCI with the Jaccard formula outperforms baselines, where the values of Accuracy@ 1, 3, and 5 are 347, 369, and 393, achieving the maximum 137.67%, 67.73%, and 47.74% improvement against baselines, respectively. Besides, we perform ablation analysis to reveal the effectiveness of features utilized in this study.

Mutation-based fault localization has shown promising accuracy in localizing faults due to its finer analysis granularity. However, the effectiveness is limited when dealing with diverse real-world systems and actual faults, which results from its inflexible suspiciousness calculation and oversimplification of information. In this work, we propose a novel Mutation-Based Fault Localization technique, GMBFL, which utilizes Graph representation to achieve multi-information cooperation to improve fault localization. GMBFL comprises two key components: a fine-grained graph-based representation to fully utilize the information of the program, and an effective suspiciousness measure using the graph neural network to learn useful features from the graph. We evaluate GMBFL on 243 real faulty programs from Defects4J. The experimental results show that GMBFL can surpass both the state-of-the-art learning-based fault localization technique and 70 commonly used SBFL and MBFL techniques. In particular, GMBFL localizes 125 faults within TOP-1 whereas the best baseline technique can at most localize 109 faults within TOP-1.

Based on developer needs and usage scenarios, API (Application Programming Interface) recommendation is the process of assisting developers in finding the required API among numerous candidate APIs. Previous studies mainly modeled API recommendation as the recommendation task, which can recommend multiple candidate APIs for the given query, and developers may not yet be able to find what they need. Motivated by the neural machine translation research domain, we can model this problem as the generation task, which aims to directly generate the required API for the developer query. After our preliminary investigation, we find the performance of this intuitive approach is not promising. The reason is that there exists an error when generating the prefixes of the API. However, developers may know certain API prefix information during actual development in most cases. Therefore, we model this problem as the automatic completion task and propose a novel approach APICom based on prompt learning, which can generate API related to the query according to the prompts (i.e., API prefix information). Moreover, the effectiveness of APICom highly depends on the quality of the training dataset. In this study, we further design a novel gradient-based adversarial training method ATCom for data augmentation, which can improve the normalized stability when generating adversarial examples. To evaluate the effectiveness of APICom, we consider a corpus of 33k developer queries and corresponding APIs. Compared with the state-of-the-art baselines, our experimental results show that APICom can outperform all baselines by at least 40.02%, 13.20%, and 16.31% in terms of the performance measures EM@1, MRR, and MAP. Finally, our ablation studies confirm the effectiveness of our component setting (such as our designed adversarial training method, our used pre-trained model, and prompt learning) in APICom.

Previous studies have been conducted on software vulnerability (SV) assessment at the code-based level, especially the function level. However, a key limitation of these studies is that they do not consider the structure information (e.g., control dependency and data dependency) of a vulnerable function, which is crucial for understanding SVs and assigning priority for fixing. In this study, we propose a flow-sensitive, multitask, and function-level vulnerability assessment method named FVA, which considers both global structure information and local semantic information. More specifically, FVA considers two types of flow information extracted from the control dependence graph and the data dependence graph. Meanwhile, FVA also considers the deep semantic information of the statement as well as its various types of contexts (i.e., surrounding context and program slicing context). We evaluate the effectiveness of FVA on the large-scale dataset (4,467 functions) by comparing it with four state-of-the-art baselines in terms of five performance measures. The experimental results indicate that FVA outperforms these baselines by a significant margin. More precisely, on average, FVA obtains 0.795 of F1-score and 0.727 of MCC, which improves baselines by 5%-14% and 8%-20%, respectively.

Question posts with low-quality titles often discourage potential answerers in Stack Overflow. In previous studies, researchers mainly focused on directly generating question titles by analyzing the contents of the posts. However, the quality of the generated titles is still limited by the information available in the post contents. A more effective way is to provide accurate completion suggestions when developers compose titles. Inspired by this idea, we are the first to study the problem of automatic question title completion for Stack Overflow and then propose a novel approach QTC4SO. Specifically, we first preprocess the gathered post titles to form incomplete titles (i.e., tip information provided by developers) for simulating the scene of this task. Then we construct the multi-modal input by concatenating the incomplete title with the post’s contents (i.e., the problem description and the code snippet). Later, we adopt multi-task learning to the question title completion task for multiple programming languages. Finally, we adopt a pre-trained model T5 to learn the title completion patterns automatically. To evaluate the effectiveness of QTC4SO, we gathered 164,748 high-quality posts from Stack Overflow by covering eight popular programming languages. Our empirical results show that compared with the approaches of directly generating question titles, our proposed approach QTC4SO is more practical in automatic and human evaluation. Therefore, our study provides a new direction for automatic question title generation and we hope more researchers can pay attention to this problem in the future.

Just-In- Time defect prediction models can identify defect-inducing commits at check-in time and many approaches are proposed with remarkable performance. However, these approaches still have a few limitations which affect their effectiveness and practical usage: (1) partially using semantic information or structure information of code, (2) coarsely providing results to a commit (buggy or clean), and (3) independently investigating the defect prediction model and defect repair model. In this study, to handle the aforementioned limitations, we propose a unified defect prediction and repair framework named COMPDEFECT,which can identify whether a changed function inside a commit is defect-prone, categorize the type of defect, and repair such a defect automatically if it falls into several scenarios, e.g., defects with single statement fixes, or those that match a small set of defect templates. Technically, the first two tasks in COMPDEFECT are treated as a multiclass classification task, while the last task is treated as a sequence generation task. To verify the effectiveness of COMPDEFECT, we first build a large-scale function-level dataset (i.e., 21,047) named Function-SStuBs4J and then compare COMPDEFECT with tens of state-of-the-art (SOTA) approaches by considering five performance measures. The experimental results indicate that COMPDEFECT outperforms all SOTAs with a substantial improvement in three tasks separately. Moreover, the pipeline experimental results also indicate the feasibility of COMPDEFECT to unify three tasks in a model.

The code comment generation task is designed for developers to understand programs more quickly during development and maintenance.However, the existing automatic code comment generation models can not generate valuable comments for developers.It is necessary to explore a technology that can optimize the performance of code comment generation models without changing the model.We consider adversarial training as the experimental object, which can improve the robustness and generalization of the model.We present a large-scale study to experimentally validate the performance of gradient-based adversarial training methods in the code comment generation task.The results show that adversarial training can improve the model performance by generating adversarial examples without changing the model.Our empirical study can provide a new perspective for researchers to improve the performance of code comment generation models.

Code comments are essential for software development and maintenance, as they provide natural language descriptions of the code that help developers understand the program and reduce the time spent on comprehension. However, writing code comments can be tedious and time-consuming, and many software projects lack comprehensive and up-to-date comments, which can impair the readability and maintainability of programs.

With the applications of intelligent systems in areas (such as self-driving cars, robotics, and smart cities), the impact of these intelligent systems’ defects cannot be ignored. For example, in a recent report, the self-driving car collided with another self-driving car because it incorrectly identified a roadblock. Therefore, it is necessary to conduct adequate testing of intelligent systems to avoid dangerous behaviors as much as possible. However, due to the particularity of its own structure, the low efficiency, and the high cost of manual collection the large-scale test cases, it is important and challenging to design tools to test the adequacy of intelligent systems.To overcome the above problems, we propose an intelligent system test adequacy evaluation tool ISTA. ISTA implements the automatic generation and optimization of test cases based on coverage analysis, which can improve the test adequacy of the intelligent system while expanding the dataset. To evaluate the usefulness of our developed tool, we analyze the application of ISTA on the five-layer fully-connected dnn model and german credit dataset (text data type) for binary classification as well as on the Rambo model and hmb dataset (image data type) for self-driving car. The evaluation results show that the test dataset is expanded and the models are more fully tested after ISTA’s test case generation and optimization for both text and image data types, with a corresponding increase in the average 80% coverage criteria used.

Abstract As agile software development and extreme programing have become increasingly popular, continuous integration (CI) has become a widely used collaborative work method. However, it is common to make changes frequently to a project during CI. If existing testing methods are applied to CI directly, it will be difficult to make testing resources focus on changes generated by CI, which results in insufficient testing for changes. To solve this problem, we propose a fuzz testing method for CI. First, differential analysis is performed to determine the change points generated during CI, change points are added to the taint source set, and static analysis is conducted to calculate the distances between each basic block and the taint sources. Then, the project under test is instrumented according to the distances. During fuzz testing, testing resources are allocated based on seed coverage to test the change points effectively. Using the proposed methods, we implement CIDFuzz as a prototype tool, and experiments are conducted on four open‐source projects that use CI. Experimental results show that, compared with AFL and AFLGo, CIDFuzz can reduce the time costs of covering change points up to 39.59% and 41.64%, respectively. Also, CIDFuzz can reduce the time costs of reproducing vulnerabilities up to 34.78% and 25.55%.

Automatically localizing faulty statements is a desired feature for effective learning programming. Most of the existing automated fault localization techniques are developed and evaluated on commercial or well-known open-source projects, which performed poorly on novice programs. In this paper, we propose a novel fault localization technique VsusFL (Variable-suspiciousness-based Fault Localization) for novice programs. VsusFL is inspired by simulating the manual program debugging process and takes advantage of variable value sequences. VsusFL can trace variable value changes, determine whether the intermediate state of the variables is correct, and report the potential faulty statements for novice programs. This paper presents the implementation of VsusFL and conducts empirical studies on 422 real faulty novice programs. Experimental results show that VsusFL performs much better than Grace, ANGELINA, VSBFL, Spectrum-Based Fault Localization (SBFL), and Variable-based Fault Localization (VFL) in terms of T O P -1, T O P -3, and T O P -5 metrics. Specifically, VsusFL can localize 90%, 35% and 9% more faulty statements than the best-performing baseline Grace. Moreover, We analyze the correlation between VsusFL and other techniques and find a weak correlation since they perform well on different programs, indicating the potential to further enhance fault localization performance through strategic integration of VsusFL with other methods.

Abstract Compilation errors are unavoidable during the debugging process of novice students. Compiler error messages can help novices to localize and remove errors, but these messages are difficult to understand for students. Previous studies have investigated the compilation error categorization by analyzing compiler error messages, but the categorization cannot cover all kinds of errors, which limits the evaluation of compilation error studies. Therefore, a comprehensive categorization for compilation errors is needed for evaluating the performance of models or tools related to compilation errors. In this study, we first propose a new compilation error categorization, which is based on the smallest unit of the program, tokens. The experiments on 29,573 programs from three datasets show that our proposed compilation error categorization can cover more types of errors and the distribution of the error categorization are significantly different between the datasets. Then, based on our proposed categorization, we develop a neural network model CLACER (CLAssification of Compilation ERrors) for predicting the compilation errors. The results indicate that CLACER can improve the compiler's error localization accuracy and predicts the compilation error effectively. Moreover, based on the proposed categorization, we conduct empirical studies to evaluate the performance of three repairing tools (i.e., DeepFix, RLAssist, and MACER). The comparison results illustrate that DeepFix and RLAssist can fix more errors in the category of delimiter than errors in other categories. Furthermore, MACER performs better than DeepFix and RLAssist because it has a sufficient repairing pattern set for the errors. We also provide some suggestions for improving the repairing tools in the future.

Abstract Identifying the location of faults in real‐world programs is one of the costly processes during software debugging. To reduce the debugging effort, various fault localization techniques have been proposed in recent years. Spectrum‐based fault localization (SBFL) is one kind of widely investigated fault localization technique. Most SBFL techniques first calculate the suspiciousness of program elements (such as statements, methods) to be faulty using the coverage information and execution results of tests. Then a rank list of program elements is generated according to their suspiciousness. However, some SBFL techniques only consider the binary coverage information (i.e., whether the program element is covered) but ignore some of the tests' running behaviors, such as the execution frequency when faults occur in the iteration entities or loop bodies, which are more likely to be faulty followed the propagation‐infection‐execution model. The execution frequency based techniques only replace the feature items of the existing formula limiting their effectiveness in fault localization. In this article, we propose a fault localization technique, class reduction and method call frequency (CRMF), which utilizes mutation analysis and information retrieval techniques. In particular, CRMF first uses mutation analysis to identify and reduce the classes, in which the program elements with a low probability of being faulty. Then we propose a new suspiciousness formula that applies information retrieval and considers method call frequency. To evaluate the effectiveness of CRMF, we conduct empirical studies on 264 real‐world programs from the Defects4J benchmark. Final results show that CRMF outperforms the statement frequency based technique FLSF and SBFL techniques (i.e., Ochiai, OP2, Tarantula, and Dstar) in both single‐fault programs and multiple‐fault programs. Specifically, CRMF can rank 29, 74, and 112 faults at the top 1, 3, 5 ranks and achieve a higher mean reciprocal rank for single‐fault programs and multiple‐fault programs. Finally, we discuss the essence of CRMF and analyze its effectiveness on multi‐fault programs in detail.

Test case prioritization (TCP) has been widely studied in regression testing, which aims to optimize the execution order of test cases so as to detect more faults earlier. TCP has been divided into white-box test case prioritization (WTCP) and black-box test case prioritization (BTCP) . WTCP can achieve better prioritization effectiveness by utilizing source code information, but is not applicable in many practical scenarios (where source code is unavailable, e.g., outsourced testing). BTCP has the benefit of not relying on source code information, but tends to be less effective than WTCP. That is, both WTCP and BTCP suffer from limitations in the practical use. To improve the practicability of TCP, we aim to explore better BTCP, significantly bridging the effectiveness gap between BTCP and WTCP. In this work, instead of statically analyzing test cases themselves in existing BTCP techniques, we conduct the first study to explore whether this goal can be achieved via log analysis. Specifically, we propose to mine test logs produced during test execution to more sufficiently reflect test behaviors, and design a new BTCP framework (called LogTCP), including log pre-processing, log representation, and test case prioritization components. Based on the LogTCP framework, we instantiate seven log-based BTCP techniques by combining different log representation strategies with different prioritization strategies. We conduct an empirical study to explore the effectiveness of LogTCP. Based on 10 diverse open-source Java projects from GitHub, we compared LogTCP with three representative BTCP techniques and four representative WTCP techniques. Our results show that all of our LogTCP techniques largely perform better than all the BTCP techniques in average fault detection, to the extent that they become competitive to the WTCP techniques. That demonstrates the great potential of logs in practical TCP.

Automated code comment generation technologies can help developers understand code intent, which can significantly reduce the cost of software maintenance and revision. The latest studies in this field mainly depend on deep neural networks, such as convolutional neural networks and recurrent neural network. However, these methods may not generate high-quality and readable code comments due to the long-term dependence problem, which means that the code blocks used to summarize information are far from each other. Owing to the long-term dependence problem, these methods forget the previous input data’s feature information during the training process. In this article, to solve the long-term dependence problem and extract both the text and structure information from the program code, we propose a novel improved-Transformer-based comment generation method, named SeTransformer. Specifically, the SeTransformer utilizes the code tokens and an abstract syntax tree (AST) of programs to extract information as the inputs, and then, it leverages the self-attention mechanism to analyze the text and structural features of code simultaneously. Experimental results based on public corpus gathered from large-scale open-source projects show that our method can significantly outperform five state-of-the-art baselines (such as Hybrid-DeepCom and AST-attendgru). Furthermore, we also conduct a questionnaire survey for developers, and the results show that the SeTransformer can generate higher quality comments than those of other baselines.

In Stack Overflow, developers may not clarify and summarize the critical problems in the question titles due to a lack of domain knowledge or poor writing skills. Previous studies mainly focused on automatically generating the question titles by analyzing the posts’ problem descriptions and code snippets. In this study, we aim to improve title quality from the perspective of question title reformulation and propose a novel approachQETRAmotivated by the findings of our formative study. Specifically, by mining modification logs from Stack Overflow, we first extract title reformulation pairs containing the original title and the reformulated title. Then we resort to multi-task learning by formalizing title reformulation for each programming language as separate but related tasks. Later we adopt a pre-trained model T5 to automatically learn the title reformulation patterns. Automated evaluation and human study both show the competitiveness ofQETRAafter compared with six state-of-the-art baselines. Moreover, our ablation study results also confirm that our studied question title reformulation task is more practical than the direct question title generation task for generating high-quality titles. Finally, we develop a browser plugin based onQETRAto facilitate the developers to perform title reformulation. Our study provides a new perspective for studying the quality of post titles and can further generate high-quality titles.

Testing multi-threaded programs is challenging due to the enormous space of thread interleavings. Recently, a code coverage criterion for multi-threaded programs called MAP-coverage has been proposed and shown to be effective for testing concurrent programs. Existing approaches for achieving high MAP-coverage are based on random testing with simple heuristics, which is ineffective in systematically triggering rare thread interleavings. In this study, we propose a novel approach called pattern constraint reduction (PCR), which employs optimized constraint solving to generate thread interleavings for high MAP-coverage. The idea is to iteratively encode and solve path conditions to generate thread interleavings which are guaranteed to improve MAP-coverage. Furthermore, we effectively apply interpolation techniques to reduce the efforts of constraint solving by avoiding solving infeasible constraints. The experiment results on 20 benchmark programs show that our approach complements existing random testing based approaches when there are rare failure-inducing interleaving in the whole search space. Specifically, PCR finds concurrency bugs faster in 18 out of 20 programs, with an average speedup of 4.2x and a maximum speedup of 11.4x.

Developers use shell commands for many tasks, such as file system management, network control, and process management. Bash is one of the most commonly used shells and plays an important role in Linux system development and maintenance. Due to the language flexibility of Bash code, developers who are not familiar with Bash often have difficulty understanding the purpose and functionality of Bash code. In this study, we study Bash code comment generation problem and proposed an automatic method BASHEXPLAINER based on two-stage training strategy. In the first stage, we train a Bash encoder by fine-tuning CodeBERT on our constructed Bash code corpus. In the second stage, we first retrieve the most similar code from the code repository for the target code based on semantic and lexical similarity. Then we use the trained Bash encoder to generate two vector representations. Finally, we fuse these two vector representations via the fusion layer and generate the code comment through the decoder. To show the competitiveness of our proposed method, we construct a high-quality corpus by combining the corpus shared in the previous NL2Bash study and the corpus shared in the NLC2CMD competition. This corpus contains 10,592 Bash codes and corresponding comments. Then we selected ten baselines from previous studies on automatic code comment generation, which cover information retrieval methods, deep learning methods, and hybrid methods. The experimental results show that in terms of the performance measures BLEU-3/4, METEOR, and ROUGR-L, BASHEXPLAINER can outperform all baselines by at least 8.75%, 9.29%, 4.77% and 3.86%. Then we design ablation experiments to show the component setting rationality of BASHEXPLAINER. Later, we conduct a human study to further show the competitiveness of BASHEXPLAINER. Finally, we develop a browser plug-in based on BASHEXPLAINER to facilitate the understanding of the Bash code for developers.

With the development of deep learning and natural language processing techniques, the performance of many source code-related tasks can be improved by using pre-trained models. Of these pre-trained models, CodeBert is a bi-modal pre-trained model for programming languages and natural languages, which has been successfully used in current source code-related tasks. These previous studies mainly use the output vector of CodeBert’s last layer as the code semantic representation for fine-tuning downstream source code-related tasks. However, this setting may miss the valuable representational information, which may be captured by other layers of CodeBert.

Mutation-Based Fault Localization (MBFL) is one of the most widely studied techniques. MBFL adopts mutation analysis to generate mutants for revealing potential faults in the program. Previous studies proposed approaches to optimize MBFL in terms of efficiency and accuracy. However, these approaches ignored the difference of mutants on correct entities (such as statements) and faulty entities, which we refer this kind of difference as mutant bias. In this study, we identify and analyze the impact of mutant bias on MBFL. We find that the mutant bias may introduce effects to statement suspiciousness and negatively influence the fault localization accuracy of MBFL. To mitigate the mutant bias, we propose Delta4Ms, a model that captures the mutant bias from the mutants of the same statements. Then the real suspiciousness is obtained by removing the bias from the practical suspiciousness. To evaluate the performance of our proposed method, we conduct experimental studies on 320 real-world programs from Codeflaws. The experimental results show that Delta4Ms improves the fault localization accuracy of MBFL. Besides, Delta4Ms outperforms the state-of-the-art SBFL and three MBFL techniques significantly. Moreover, Delta4Ms ranks 94 and 161 of the target faults within the top-5 suspicious statements in single-fault and multiple-fault programs, respectively.

The task of a deep learning (DL) program is to train a model with high precision and apply it to different scenarios. A DL program often involves massive numerical calculations. Therefore, the robustness and stability of the numerical calculations are dominant in the quality of DL programs. Indeed, numerical bugs are common in DL programs, producing NaN (Not-a-Number) and INF (Infinite). A numerical bug may render the DL models inaccurate, causing the DL applications unusable. In this work, we conduct the first empirical study on numerical bugs in DL programs by analyzing the programs implemented on the top of two popular DL libraries (i.e., TensorFlow and PyTorch). Specifically, We collect a dataset of 400 numerical bugs in DL programs. Then, we classify these numerical bugs into nine categories based on their root causes and summarize two findings. Finally, we provide the implications of our study on detecting numerical bugs in DL programs.

On Stack Overflow, developers can not only browse question posts to solve their programming problems but also gain expertise from the question posts to help improve their programming skills. Therefore, improving the quality of question posts in Stack Overflow has attracted the wide attention of researchers. A concise and precise title can play an important role in helping developers understand the key information of the question post, which can improve the post quality. How-ever, the quality of the generated title is not high due to the lack of professional knowledge related to their questions or the poor presentation ability of developers. A previous study aimed to automatically generate the title by analyzing the code snippets in the question post. However, this study ignored the useful information in the corresponding problem description. Therefore, we propose an approach SOTitle for automatic post title generation by leveraging the code snippets and the problem description in the question post (i.e., the multi-modal input). SOTitle follows the Transformer structure, which can effectively capture long-term dependencies through a multi-head attention mechanism. To verify the effectiveness of SOTitle, we construct a large-scale high-quality corpus from Stack Overflow, which includes 1,168,257 high-quality question posts for four popular programming languages. Experimental results show that SOTitle can significantly outperform six state-of-the-art baselines in both automatic evaluation and human evaluation. To encourage follow-up studies, we make our corpus and approach publicly available.

A shellcode is a small piece of code and it is executed to exploit a software vulnerability, which allows the target computer to execute arbitrary commands from the attacker through a code injection attack. Similar to the purpose of automated vulnerability generation techniques, the automated generation of shellcode can generate attack instructions, which can be used to detect vulnerabilities and implement defensive measures. While the automated summarization of shellcode can help users unfamiliar with shellcode and network information security understand the intent of shellcode attacks. In this study, we propose a novel approach DualSC to solve the automatic shellcode generation and summarization tasks. Specifically, we formalize automatic shellcode generation and summarization as dual tasks, use a shallow Transformer for model construction, and design a normalization method Adjust_QKNorm to adapt these low-resource tasks (i.e., insufficient training data). Finally, to alleviate the out-of-vocabulary problem, we propose a rule-based repair component to improve the performance of automatic shellcode generation. In our empirical study, we select a high-quality corpus Shellcode_IA32 as our empirical subject. This corpus was gathered from two real-world projects based on the line-by-line granularity. We first compare DualSC with six state-of-the-art baselines from the code generation and code summarization domains in terms of four performance measures. The comparison results show the competitiveness of DualSC. Then, we verify the effectiveness of the component setting in DualSC. Finally, we conduct a human study to further verify the effectiveness of DualSC.

For blockchain software systems, framework developers may introduce technical debts that application developers are not aware of. Because these technical debts can have a negative impact on software projects, we need to investigate the issue of technical debt in blockchain software systems. We wanted to investigate what types of self-introduced technical debt exist in open-source blockchain software systems, and how these technical debts are distributed. We have selected six most popular blockchain software projects from GitHub. Then the code comments from these software projects were extracted and manually labelled. Finally, the code comments were statistically analysed. We propose a new type of technical debt, resource debt, which is explicitly identified by the framework developers and requires special attention in subsequent production systems. Six types of technical debt are prevalent and there is not any algorithm debt. In addition, we find that the code comments containing technical debt are not entirely determined by task tags. SATD is prevalent in blockchain projects. There is more significant variability between different application software projects for different technical debts. The results of the study imply that for detecting SATD, deep semantic discovery models should be used, such as pre-trained models.

Fault localization is one of the most expensive activities in software debugging. Mutation-based fault localization (MBFL) is a commonly studied technique that applied mutation analysis to find the location of faults in the programs. Previous studies showed that MBFL adopted First-Order-Mutants (FOMs) that could achieve promising results in single-fault localization, but it did not perform well in multiple-fault localization. Recently, Higher-Order-Mutants (HOMs) were proposed for modeling complex faults but whether HOMs can help in fault localization is still unknown. In this paper, we investigate the performance of MBFL with FOMs and HOMs on single- and multiple-fault localization. Moreover, to study the characteristics of HOMs, we divide HOMs into three groups (i.e. Accurate HOMs, Partially accurate HOMs, and Inaccurate HOMs) by considering different mutation locations. Based on the empirical results on 186 versions of six real-world programs, we find that (1) In single-fault localization, FOMs can achieve better performance than HOMs. (2) However, in multiple-fault localization, HOMs (2-HOMs) localize more faults than FOMs. (3) Furthermore, different types of HOMs have different fault localization effectiveness, where Accurate HOMs outperform the other two HOMs categories. Therefore, the researchers should propose methods to find HOMs more useful for fault localization.

First-order mutants (FOMs) have been widely used in mutation-based fault localization (MBFL) approaches and have achieved promising results in single-fault localization scenarios (SFL-scenario). Higher-order mutants (HOMs) are proposed to simulate complex faults and can be applied in MBFL theoretically for multiple-fault localization scenarios (MFL-scenario). However, whether HOMs can improve MBFL’s performance is not investigated and the effectiveness is not thoroughly evaluated. In this empirical study, we investigate the impact of HOMs on the performance of MBFL in SFL-scenario and MFL-scenario. The experiments on two real-world benchmarks reveal that 1) 2-HOMs can help improve the MBFL performance in SFL-scenarios; 2) in MFL-scenarios, both 2-HOMs and 3-HOMs can achieve better performance than FOMs; and 3) huge computational cost cannot be ignored in the practice of HOMs. Therefore, effective methods to reduce the number of HOMs for future MBFL studies should be considered.

To improve the efficiency of the fault localization process, different automatic fault localization approaches have been proposed. Among these approaches, the spectrum-based fault localization (SBFL) approach has been widely used and studied due to its lightweight and high effectiveness. However, while the existence of coincidental correct (CC) test cases can influence the usefulness of SBFL in single-fault programs, their influence on multiple fault programs has not been thoroughly investigated. Therefore, in this article, we conduct a theoretical analysis and an empirical study to investigate the effect of CC test cases on multiple fault localization. The theoretical analysis is based on a suspiciousness calculation formula of SBFL, which divides CC test cases into three categories (specific, irrelevant, and unspecific) according to their association with a specific faulty statement. Following this analysis, we conduct an empirical study on two well-known open-source repositories (SIR and Defects4J), and the experimental results verify the correctness of our theoretical analysis. Specifically, reducing the number of specific CC test cases for a faulty statement can improve or maintain fault localization accuracy, while eliminating irrelevant CC test cases can have a negative effect. Finally, we design a CC test case identification solution based on the isolation-based multiple fault localization approach and demonstrate its effectiveness via a simulation experiment.

Cross-project defect prediction (CPDP), aiming to apply defect prediction models built on source projects to a target project, has been an active research topic. A variety of supervised CPDP methods and some simple unsupervised CPDP methods have been proposed. In a recent study, Zhouet al.found that simple unsupervised CPDP methods (i.e., ManualDown and ManualUp) have a prediction performance comparable or even superior to complex supervised CPDP methods. Therefore, they suggested that the ManualDown should be treated as the baseline when considering non-effort-aware performance measures (NPMs) and the ManualUp should be treated as the baseline when considering effort-aware performance measures (EPMs) in future CPDP studies. However, in that work, these unsupervised methods are only compared with existing supervised CPDP methods using a small subset of NPMs, and the prediction results of baselines are directly collected from the primary literatures. Besides, the comparison has not considered other recently proposed EPMs, which consider context switches and developer fatigue due to initial false alarms. These limitations may not give a holistic comparison between the supervised methods and unsupervised methods. In this paper, we aim to revisit Zhouet al.’s study. To the best of our knowledge, we are the first to make a comparison between the existing supervised CPDP methods and the unsupervised methods proposed by Zhouet al.in the same experimental setting when considering both NPMs and EPMs. We also propose an improved supervised CPDP method EASC and make a further comparison with the unsupervised methods. According to the results on 82 projects in terms of 11 performance measures, we find that when considering NPMs, EASC can achieve prediction performance comparable or even superior to unsupervised method ManualDown in most cases. Besides, when considering EPMs, EASC can statistically significantly outperform the unsupervised method ManualUp with a large improvement in terms of Cliff’s delta in most cases. Therefore, the supervised CPDP methods are more promising than the unsupervised method in practical application scenarios, since the limitation of testing resource and the impact on developers cannot be ignored in these scenarios.

Different from within-project software defect prediction (WPDP), cross-project software defect prediction (CPDP) does not require sufficient training data and can help developers in the early stages of software development. Recent studies tried to learn semantic features for CPDP by feeding neural networks with abstract syntax tree (AST) token vectors. However, the ASTs directly parsed from software modules usually have complex structures, which are reflected on more nodes and deeper size, and the transfer learning is not regularly adopted to further reduce the data distribution difference between the source project and the target project. To solve these problems, we aim to joint learn the statement level trees (SLT) and alleviate data distribution difference with maximum mean discrepancy (MMD) to improve defect prediction performance on CPDP. Specifically, we propose a novel cross-project defect prediction method S2LMMD via statement semantic learning and MMD. We first construct the SLT by splitting the original AST on specified node. Then we generate more effective semantic features by learning of sequence embedding with Bi-GRU neural network. Finally, a transfer loss MMD is carried out to keep more common characteristics across different project datasets to further improve CPDP performance. To verify the effectiveness of our proposed method, we conducted experiments on ten widely used open-source projects and evaluated the experimental performance by using AUC measures. Our empirical results show that our proposed method S2LMMD can significantly outperform eight state-of-the-art baselines. In addition, for semantic learning, SLT has a higher influence on CPDP, while MMD is of great significance in transfer learning.

Pseudo-code written by natural language is helpful for novice developers' program comprehension. However, writing such pseudo-code is time-consuming and laborious. Motivated by the research advancements of sequence-to-sequence learning and code semantic learning, we propose a novel deep pseudo-code generation method DeepPseudo via code feature extraction and Transformer. In particular, DeepPseudo utilizes a Transformer encoder to perform encoding for source code and then use a code feature extractor to learn the knowledge of local features. Finally, it uses a pseudo-code generator to perform decoding, which can generate the corresponding pseudo-code. We choose two corpora (i.e., Django and SPoC) from real-world large-scale projects as our empirical subjects. We first compare DeepPseudo with seven state-of-the-art baselines from pseudo-code generation and neural machine translation domains in terms of four performance measures. Results show the competitiveness of DeepPseudo. Moreover, we also analyze the rationality of the component settings in DeepPseudo.

Compilation errors happen during the debugging process of novice students. Compiler error messages help novices to localize and remove errors, but these messages are difficult to understand for students. Some computing education researchers analyzed the compiler error messages generated by novice’s attempts to compile their programs. However, some important questions remain open. For example, the existing compilation error category cannot cover all programs due to the simple static analysis and program repair patterns. And existing prediction models for classifying compilation errors are unsatisfactory because of the inappropriate neural networks. In this paper, we first propose a new category of compilation error based on the program tokens, which is the smallest unit of the program. Then we develop a neural network model CLACER (ClAssification of Compilation ERrors) based on TextCNN. CLACER performs better on extracting semantic features and statistical features from compiler error messages. To verify the effectiveness of our proposed category and corresponding method CLACER, we choose 16,926 student programs as our experimental subjects. Final experimental results indicate that our proposed classification category covers 16.5% more programs than the state-of-the-art category TEGCER. Moreover, CLACER improves the compiler’s localization effectiveness and with a 4.25% improvement on the TEGCER category. Further analysis shows that CLACER has a promising prediction performance for different error classes, and TextCNN is more suitable for constructing the compilation error classification model.

As a popular Q&A site for programming, Stack Overflow is a treasure for developers. However, the amount of questions and answers on Stack Overflow make it difficult for developers to efficiently locate the information they are looking for. There are two gaps leading to poor search results: the gap between the user's intention and the textual query, and the semantic gap between the query and the post content. Therefore, developers have to constantly reformulate their queries by correcting misspelled words, adding limitations to certain programming languages or platforms, etc. As query reformulation is tedious for developers, especially for novices, we propose an automated software-specific query reformulation approach based on deep learning. With query logs provided by Stack Overflow, we construct a large-scale query reformulation corpus, including the original queries and corresponding reformulated ones. Our approach trains a Transformer model that can automatically generate candidate reformulated queries when given the user's original query. The evaluation results show that our approach outperforms five state-of-the-art baselines, and achieves a 5.6% to 33.5% boost in terms of ExactMatch and a 4.8% to 14.4% boost in terms of GLEU.

Identifying and fixing bug reports with tactical vul-nerabilities in a timely and accurate manner is essential to ensure the security of the software architecture. Manually identifying the bug reports with tactical vulnerabilities is labor-intensive and challenging. This paper presents Itactivul, an approach to automatically identify bug reports with tactical vulnerabilities and recommend their tactical categories to guide the fix. Unlike the existing security bug report prediction approach, we are the first attempt to use deep learning to mine discriminative tactical text features only from the vulnerability descriptions of the National Vulnerability Database (NVD) and apply them to identify bug reports with tactical vulnerabilities. We evaluate Itactivul on three bug reports datasets gathered from three large-scale open-source projects, including Chromium, PHP, and Thunderbird. The experimental results show that Itactivul outperforms baselines by an average of 8.88 %, 13.58 %, and 6.61 % in the F1-score of three datasets, respectively. To improve the explainability of the features mined by Itactivul, we manually analyze the high-weight phrases extracted by using attention backtracking. The results show that Itactivul can mine key and potential tactical vulnerabilities text features.

Concurrency programs often induce buggy results due to the unexpected interaction among threads. The detection of these concurrency bugs costs a lot because they usually appear under a specific execution trace. How to virtually explore different thread schedules to detect concurrency bugs efficiently is an important research topic. Many techniques have been proposed, including lightweight techniques like adaptive randomized scheduling (ARS) and heavyweight techniques like maximal causality reduction (MCR). Compared to heavyweight techniques, ARS is efficient in exploring different schedulings and achieves state-of-the-art performance. However, it will lead to explore large numbers of redundant thread schedulings, which will reduce the efficiency. Moreover, it suffers from the “cold start” issue, when little information is available to guide the distance calculation at the beginning of the exploration. In this work, we propose a Heuristic-Enhanced Adaptive Randomized Scheduling (HARS) algorithm, which improves ARS to detect concurrency bugs guided with novel distance metrics and heuristics obtained from existing research findings. Compared with the adaptive randomized scheduling method, it can more effectively distinguish the traces that may contain concurrency bugs and avoid redundant schedules, thus exploring diverse thread schedules effectively. We conduct an evaluation on 45 concurrency Java programs. The evaluation results show that our algorithm performs more stably in terms of effectiveness and efficiency in detecting concurrency bugs. Notably, HARS detects hard-to-expose bugs more effectively, where the buggy traces are rare or the bug triggering conditions are tricky.

Fault localization techniques have been developed for decades. Spectrum Based Fault Localization (SBFL) is a popular strategy in this research topic. However, SBFL is well known for low accuracy, mainly due to simply using a coverage matrix of program executions. In this paper, we propose a method based on graph neural network (AGFL), characterized by the adjacent matrix of the abstract syntax tree and the word vector of each program token. Referring to the Dstar, we calculate the suspiciousness of the statements and rank these statements. The experiment carried on Defects4J, a widely used benchmark, reveals that AGFL can locate 178 of the 262 studied bugs within Top-1, while state-of-the-art techniques at most locate 148 within Top-1. We also investigate the impacts of hyper-parameters (e.g., epoch and learning rate). The results show that AGFL has the best effect when the epoch is 100 and the learning rate is 0.0001. This value of epoch and learning rate increases by 66% compared to the worst on Top-1.

Code summarization aims to automatically generate functionality descriptions of code snippets. Faithful metrics are needed to measure to which degree the machine generated summaries capture the semantics of the code snippets. Most commonly used metrics in code summarization, such as BLEU -4, METEOR, and ROUGE-L, originate from machine translation and text summarization, and have constantly been found to be inconsistent with human assessment. In this paper, we propose a novel evaluation metric, Consensus-based Code Summarization Evaluation (CCSE), which assigns different semantic weights to the n-grams of the summary. We also provide an algorithm to match the n-gram pairs from the reference and candidate based on the similarities. To validate the effectiveness of our proposed metric, we collect summary pairs from two public Java datasets and calculate the correlation coefficients between CCSE and the human evaluations. The experiment results show that, compared with BLEU-4, METEOR, and ROUGE-L, CCSE is more consistent with the scores assessed by human developers.

In software engineering-related tasks (such as programming language tag prediction based on code snippets from Stack Overflow), the programming language classification for code snippets is a common task.In this study, we propose a novel method DeepSCC, which uses a fine-tuned RoBERTa model to classify the programming language type of the source code.In our empirical study, we choose a corpus collected from Stack Overflow, which contains 224,445 pairs of code snippets and corresponding language types.After comparing nine state-of-the-art baselines from the fields of source code classification and neural text classification in terms of four performance measures (i.e., Accuracy, Precision, Recall, and F1), we show the competitiveness of our proposed method DeepSCC.

There are increasing uses of deep learning (DL) compilers to generate optimized code, boosting the runtime performance of DL models on specific hardware. Like their traditional counterparts, DL compilers can generate incorrect code, resulting in unexpected model behaviors that may cause catastrophic consequences in mission-critical systems. On the other hand, the DL models processed by DL compilers differ fundamentally from imperative programs in that the program logic in DL models is implicit. As such, various characteristics of the bugs arising from traditional compilers need to be revisited in the context of DL compilers.

A flaky test is a test that both passes and fails periodically without any code changes, and its uncontrolled uncertainty will destroy the value of the test suites and even cause developers to distrust the test results. Recently, researches in the flaky test have received broad attention in the software test community to reduce the manual maintenance cost of flaky tests by developers. In this survey, we conducted comprehensive research progress on the flaky test We identified 31 relevant studies and summarized the following aspects of the flaky test: root causes and factors, analyzing the impact, detecting and classifying techniques, and fixing approaches. This survey also identifies open research challenges to be further explored in future work.

Abstract In recent years, the widespread applications of open-source software (OSS) have brought great convenience for software developers. However, it is always facing unavoidable security risks, such as open-source code defects and security vulnerabilities. To find out the OSS risks in time, we carry out an empirical study to identify the indicators for evaluating the OSS. To achieve a comprehensive understanding of the OSS assessment, we collect 56 papers from prestigious academic venues (such as IEEE Xplore, ACM Digital Library, DBLP, and Google Scholar) in the past 21 years. During the process of the investigation, we first identify the main concerns for selecting OSS and distill five types of commonly used indicators to assess OSS. We then conduct a comparative analysis to discuss how these indicators are used in each surveyed study and their differences. Moreover, we further undertake a correlation analysis between these indicators and uncover 13 confirmed conclusions and four cases with controversy occurring in these studies. Finally, we discuss several possible applications of these conclusions, which are insightful for the research on OSS and software supply chain.

Abstract Security vulnerability prediction (SVP) can construct models to identify potentially vulnerable program modules via machine learning. Two kinds of features from different points of view are used to measure the extracted modules in previous studies. One kind considers traditional software metrics as features, and the other kind uses text mining to extract term vectors as features. Therefore, gathered SVP data sets often have numerous features and result in the curse of dimensionality. In this article, we mainly investigate the impact of filter‐based ranking feature selection (FRFS) methods on SVP, since other types of feature selection methods have too much computational cost. In empirical studies, we first consider three real‐world large‐scale web applications. Then we consider seven methods from three FRFS categories for FRFS and use a random forest classifier to construct SVP models. Final results show that given the similar code inspection cost, using FRFS can improve the performance of SVP when compared with state‐of‐the‐art baselines. Moreover, we use McNemar's test to perform diversity analysis on identified vulnerable modules by using different FRFS methods, and we are surprised to find that almost all the FRFS methods can identify similar vulnerable modules via diversity analysis.

Identifying security bug reports (SBRs) accurately from a bug repository can reduce a software product’s security risk. However, the class imbalance problem exists for SBR prediction since the number of SBRs is often limited, and this issue has not been thoroughly investigated in previous studies. In our study, we choose six real-world projects of different sizes with over 120 000 bug reports in total as our empirical subjects. We first analyze the impact of the class imbalance issue on SBR prediction and confirm its negative impact on prediction performance. Then we perform a comparative study of six state-of-the-art class rebalancing methods combined with five popular classification algorithms for SBR prediction. By comparing with the baseline method Farsec, using the class rebalancing methods can improve the performance in 78% of cases in the worst case. Moreover, the combination of the Rose and random forest classification algorithm can construct the model with the best performance, which increases the performance by 267% in the best case and 75% on average in terms ofF1-score. Finally, we summarize eight main findings based on our empirical studies’ results, which can provide guidelines for choosing appropriate class rebalancing methods and classifiers for SBR prediction in practice.

Informal discussions on social platforms (e.g., Stack Overflow, CodeProject) have accumulated a large body of programming knowledge in the form of natural language text. Natural language process (NLP) techniques can be utilized to harvest this knowledge base for software engineering tasks. However, consistent vocabulary for a concept is essential to make an effective use of these NLP techniques. Unfortunately, the same concepts are often intentionally or accidentally mentioned in many different morphological forms (such as abbreviations, synonyms and misspellings) in informal discussions. Existing techniques to deal with such morphological forms are either designed for general English or mainly resort to domain-specific lexical rules. A thesaurus, which contains software-specific terms and commonly-used morphological forms, is desirable to perform normalization for software engineering text. However, constructing this thesaurus in a manual way is a challenge task. In this paper, we propose an automatic unsupervised approach to build such a thesaurus. In particular, we first identify software-specific terms by utilizing a software-specific corpus (e.g., Stack Overflow) and a general corpus (e.g., Wikipedia). Then we infer morphological forms of software-specific terms by combining distributed word semantics, domain-specific lexical rules and transformations. Finally, we perform graph analysis on morphological relations. We evaluate the coverage and accuracy of our constructed thesaurus against community-cumulated lists of software-specific terms, abbreviations and synonyms. We also manually examine the correctness of the identified abbreviations and synonyms in our thesaurus. We demonstrate the usefulness of our constructed thesaurus by developing three applications and also verify the generality of our approach in constructing thesauruses from data sources in other domains.

Unlike code errors, the presence of code smell often does not affect the behavior of the software system, but it will cause quality problems in terms of readability, understandability, and efficiency. To improve the software quality and reduce the maintenance costs, the developers need to detect code smells rapidly and make corresponding code refactoring. In code smell detection, recently, machine learning-based methods become more prevalent and can overcome the shortcomings of the heuristic-based methods, which mainly rely on manually designed rules. However, to our best knowledge, there is little research to analyze whether using hyper-parameter optimization can improve the performance of machine learning-based methods. In this study, we mainly focus on two classical code smells (i.e., Data Class and Feature Envy). First, we consider four optimizers for hyper-parameter optimization, and six commonly used classifiers for machine-learning-based methods. Second, we use AUC as the performance measure to evaluate the performance of constructed models. Based on final empirical results, we find that (1) Using hyper-parameter optimization can significantly improve the performance of code smell detection. (2) Differential evolution (DE) optimizer can achieve better performance than the other three optimizers when using the random forest classifier. (3) We can further improve the performance of code smell detection when performing parameter optimization on the DE optimizer.

Abstract Cross‐project defect prediction (CPDP) is needed when the target projects are new projects or the projects have less training data, since these projects do not have sufficient historical data to build high‐quality prediction models. The researchers have proposed many CPDP methods, and previous studies have conducted extensive comparisons on the performance of different CPDP methods. However, to the best of our knowledge, it remains unclear whether different CPDP methods can identify the same defective modules, and this issue has not been thoroughly explored. In this article, we select 12 state‐of‐the‐art CPDP methods, including eight supervised methods and four unsupervised methods. We first compare the performance of these methods in the same experiment settings on five widely used datasets (ie, NASA, SOFTLAB, PROMISE, AEEEM, and ReLink) and rank these methods via the Scott‐Knott test. Final results confirm the competitiveness of unsupervised methods. Then we perform diversity analysis on defective modules for these methods by using the McNemar test. Empirical results verify that different CPDP methods may lead to difference in the modules predicted as defective, especially when the comparison is performed between the supervised methods and unsupervised methods. Finally, we also find there exist a certain number of defective modules, which cannot be correctly identified by any of the CPDP methods or can be correctly identified by only one CPDP method. These findings can be utilized to design more effective methods to further improve the performance of CPDP.

Security vulnerability prediction (SVP) can identify potential vulnerable modules in advance and then help developers to allocate most of the test resources to these modules. To evaluate the performance of different SVP methods, we should take the security audit and code inspection into account and then consider effort-aware performance measures (such as ACC and Popt). However, to the best of our knowledge, the effectiveness of different SVP methods has not been thoroughly investigated in terms of effort-aware performance measures. In this article, we consider 48 different SVP methods, of which 36 are supervised methods and 12 are unsupervised methods. For the supervised methods, we consider 34 software-metric-based methods and two text-mining-based methods. For the software-metric-based methods, in addition to a large number of classification methods, we also consider four state-of-the-art methods (i.e., EALR, OneWay, CBS, and MULTI) proposed in recent effort-aware just-in-time defect prediction studies. For text-mining-based methods, we consider the Bag-of-Word model and the term-frequency-inverse-document-frequency model. For the unsupervised methods, all the modules are ranked in the ascendent order based on a specific metric. Since 12 software metrics are considered when measuring extracted modules, there are 12 different unsupervised methods. To the best of our knowledge, over 40 SVP methods have not been considered in previous SVP studies. In our large-scale empirical studies, we use three real open-source web applications written in PHP as benchmark. These three web applications include 3466 modules and 223 vulnerabilities in total. We evaluate these SVP methods both in the within-project SVP scenario and the cross-project SVP scenario. Empirical results show that two unsupervised methods [i.e., lines of code (LOC) and Halstead's volume (HV)] and four recently proposed state-of-the-art supervised methods (i.e., MULTI, OneWay, CBS, and EALR) can achieve better performance than the other methods in terms of effort-aware performance measures. Then, we analyze the reasons why these six methods can achieve better performance. For example, when using 20% of the entire efforts, we find that these six methods always require more modules to be inspected, especially for unsupervised methods LOC and HV. Finally, from the view of practical vulnerability localization, we find that all the unsupervised methods and the OneWay method have high false alarms before finding the first vulnerable module. This may have an impact on developers' confidence and tolerance, and supervised methods (especially MULTI and text-mining-based methods) are preferred.

Cross-project defect prediction (CPDP) as a means to focus quality assurance of software projects was under heavy investigation in recent years. In this paper, we propose a novel CPDP approach via deep learning. In particular, we model each program module via simplified abstract syntax tree (S-AST). For each node in S-AST, only the project-independent node type is remained and other project-specific information (such as name of variable and method) is ignored, so that the modeling method is project-independent and suitable for CPDP issue. Then we extract token sequences from program modules modeled as S-AST. In addition, to construct meaningful vector representations for token sequences, we propose a novel unsupervised embedding method ASTToken2Vec, which learns semantic information from S-AST's natural structure. Finally, we use BLSTM (bi-directional long short-term memory) based neural network to automatically learn semantic features from vectorized token sequences and construct CPDP models. In our empirical studies, 10 real large-scale open source Java projects are chosen as our empirical subjects. Final results show that our proposed CPDP approach can perform significantly better than 5 state-of-the-art CPDP baselines in terms of AUC.

Concurrent programs must be thoroughly tested, as concurrency bugs are notoriously hard to detect. Code coverage criteria can be used to quantify the richness of a test suite (e.g., whether a program has been tested sufficiently) or provide practical guidelines on test case generation (e.g., as objective functions used in program fuzzing engines). Traditional code coverage criteria are, however, designed for sequential programs and thus ineffective for concurrent programs. In this work, we introduce a novel code coverage criterion for testing thread-safe classes called MAP-coverage (short for memory-access patterns). The motivation is that concurrency bugs are often correlated with certain memory-access patterns, and thus it is desirable to comprehensively cover all memory-access patterns. Furthermore, we propose a testing method for maximizing MAP-coverage. Our method has been implemented as a self-contained toolkit, and the experimental results on 20 benchmark programs show that our toolkit outperforms existing testing methods. Lastly, we show empirically that there exists positive correlation between MAP-coverage and the effectiveness of a set of test executions.

Software defect prediction can make software quality assurance (SQA) process more efficient, economic and targeted.Previous studies mainly focused on classifying software modules as defect-prone or not.However, prediction the number of defects for a new software module is rarely investigated.Moreover, these studies built models independently for each project, which may ignore the relatedness among multiple projects.To effectively utilize the relatedness, we propose a novel approach MPR (multiproject regression) for SDNP (software defect number prediction).To verify the effectiveness of MPR, we perform experimental studies on 30 real-world projects and compare our approach with 6 state-of-the-art baselines (i.e., LR, NNR, SVR, DTR, BRR and DBR).AAE (Average absolute error) and ARE (average relative error) performance measures are used to evaluate the performance of MPR.The results show MPR can achieve better performance in most cases, which indicates the competitiveness of MPR in the context of SDNP.

Abstract Within‐project defect prediction assumes that we have sufficient labeled data from the same project, while cross‐project defect prediction assumes that we have plenty of labeled data from source projects. However, in practice, we might only have limited labeled data from both the source and target projects in some scenarios. In this paper, we want to apply multitask learning to investigate such a new scenario. To our best knowledge, this problem (ie, both the source project and the target project have limited labeled data) has not been thoroughly investigated, and we are the first to propose a novel multitask defect prediction approach mask. mask consists of a differential evolution optimization phase and a multitask learning phase. The former phase aims to find optimal weights for shared and nonshared information in related projects (ie, the target project and its related source projects), while the latter phase builds prediction models for each project simultaneously. To verify the effectiveness of mask, we perform experimental studies on 18 real‐world software projects and compare our approach with four state‐of‐the‐art baseline approaches: single‐task learning (STL), simple combined learning (SCL), Peters filter, and Burak filter. Experimental results show that mask can achieve F1 of 0.397 and AUC of 0.608 on average with a few labeled data (ie, 10% of data). Across the 18 projects, mask can outperform baseline methods significantly in terms of F1 and AUC. Therefore, by utilizing the relatedness among multiple projects, mask can perform significantly better than the state‐of‐the‐art methods. The results confirm that mask is promising for software defect prediction when the source and target projects both have limited training data.

Software defect prediction can identify potential defective modules in advance and then provide guidances for software testers to allocate more testing resources on these modules. During the gathering process for defect prediction datasets, if multiple metrics are used to measure the program modules, it will result in curse of dimensionality. Feature selection is one of effective methods to alleviate this problem. However, designing effective feature selection methods is a great challenge. Motivated by the idea of search based software engineering, we formalize this problem as a multi-objective optimization problem, and then propose novel method MOFES. To verify the effectiveness of our proposed method, we choose PROMISE dataset gathered from real projects, and compare MOFES with some classical baseline methods. Final results show that our method has the advantages of selecting less features and achieving better prediction performance in most projects while its computational cost is acceptable.

Cross project defect prediction (CPDP) is a challenging task since the predictor built on the source projects can hardly generalize well to the target project. Previous studies have shown that both feature mapping and feature selection can alleviate the differences between the source and target projects. In this paper, we propose a novel method FeSCH (Feature Selection using Clusters of Hybrid-data). In particular it includes two phases. The first is the feature clustering phase, which uses a density-based clustering method DPC to group highly co-related features into clusters. The second is the feature selection phase, which selects beneficial features from each cluster. We design three ranking strategies to choose appropriate features. During the empirical studies, we design experiments based on real-world software projects, and evaluate the prediction performance of FeSCH by analyzing the influence of ranking strategies. The experimental results show that FeSCH can outperform three baseline methods (i.e., WPDP, ALL, and TCA+) in most cases, and its performance is independent of the used classifiers.

GitHub, one of the largest social coding platforms, fosters a flexible and collaborative development process. In practice, developers in the open source software platform need to find projects relevant to their development work to reuse their function, explore ideas of possible features, or analyze the requirements for their projects. Recommending relevant projects to a developer is a difficult problem considering that there are millions of projects hosted on GitHub, and different developers may have different requirements on relevant projects. In this paper, we propose a scalable and personalized approach to recommend projects by leveraging both developers' behaviors and project features. Based on the features of projects created by developers and their behaviors to other projects, our approach automatically recommends top N most relevant software projects to developers. Moreover, to improve the scalability of our approach, we implement our approach in a parallel processing frame (i.e., Apache Spark) to analyze large-scale data on GitHub for efficient recommendation. We perform an empirical study on the data crawled from GitHub, and the results show that our approach can efficiently recommend relevant software projects with a relatively high precision fit for developers' interests.

Software fault prediction is a valuable exercise in software quality assurance to best allocate limited testing resources. Classification is one of the effective methods for software fault prediction. The classification models are trained based on the datasets obtained by mining software historical repositories. However, the performance of the models depends on the quality of datasets. In this paper, we propose a novel two-stage data preprocessing approach which incorporates both feature selection and instance reduction. Specifically, in the feature selection stage, we first perform relevance analysis, and then propose a threshold-based clustering method, called novel threshold-based clustering algorithm, to conduct redundancy control. In the instance reduction stage, we apply random under-sampling to keep the balance between the faulty and non-faulty instances. In empirical studies, we chose datasets from real-world software projects, such as Eclipse and NASA. Then we compared our approach with some classical baseline methods, and further investigated the influencing factors in our approach. The final results demonstrate the effectiveness of our approach, and provide a guideline for achieving cost-effective data preprocessing when using our two-stage approach.

Noises are inevitable when mining software archives for software fault prediction. Although some researchers have investigated the noise tolerance of existing feature selection methods, few studies focus on proposing new feature selection methods with a certain noise tolerance. To solve this issue, we propose a novel method FECS (FEature Clustering with Selection strategies). This method includes two phases: a feature clustering phase and a feature selection phase with three different heuristic search strategies. During empirical studies, we choose real-world software projects, such as Eclipse and NASA and inject class level and feature level noises simultaneously to imitate noisy datasets. After using classical feature selection methods as the baseline, we confirm the effectiveness of FECS and provide a guideline of using FECS after analyzing the effects of varying either the percentage of selected features or the noise rate.

Software defect prediction can classify new software entities into either buggy or clean. However the effectiveness of existing methods is influenced by irrelevant and redundant features. In this paper, we propose a new feature selection framework FECAR using Feature Clustering And feature Ranking. This framework firstly partitions original features into k clusters based on FF-Correlation measure. Then it selects relevant features from each cluster based on FC-Relevance measure. In empirical study, we choose Symmetric Uncertainty as FF-Correlation measure, and choose Information Gain, Chi-Square, and Relief as three different FC-Relevance measures. Based on some real projects Eclipse and NASA, we implemented our framework and performed empirical studies to investigate the redundancy rate and the performance of the trained defect predictors. Final results verify the effectiveness of our proposed framework and further provide a guideline for achieving cost-effective feature selection when using our framework.

Classification based software fault prediction methods aim to classify the modules into either fault-prone or non-fault-prone. Feature selection is a preprocess step used to improve the data quality. However most of previous research mainly focus on feature relevance analysis, there is little work focusing on feature redundancy analysis. Therefore we propose a two-stage framework for feature selection to solve this issue. In particular, during the feature relevance phase, we adopt three different relevance measures to obtain the relevant feature subset. Then during the feature redundancy analysis phase, we use a cluster-based method to eliminate redundant features. To verify the effectiveness of our proposed framework, we choose typical real-world software projects, including Eclipse projects and NASA software project KC1. Final empirical result shows the effectiveness of our proposed framework.

In recent years, researchers have paid more attention to influence maximization problem. This problem is firstly defined by Domingos and Richardson as follows: finding a small set of individuals in a social network that could maximize the spread of influence under certain influence cascade model. To solve this issue, researchers proposed different algorithms. However, in all of these algorithms, the size of the chosen individuals -k, is assigned in advance. In this paper, we conduct a preliminary exploration on the relationship between the size of the chosen set and the corresponding influence spread. We propose two metrics to analyze the factor k. Then we further consider the performance metric of the chosen set, which can be described by the stability of the chosen set. Experimental results on two real social networks show the efficiency and necessity of our proposed metrics.

Combinatorial testing (also called interaction testing) is an effective specification-based test input generation technique. By now most of research work in combinatorial testing aims to propose novel approaches trying to generate test suites with minimum size that still cover all the pairwise, triple, or n-way combinations of factors. Since the difficulty of solving this problem is demonstrated to be NP-hard, existing approaches have been designed to generate optimal or near optimal combinatorial test suites in polynomial time. In this paper, we try to apply particle swarm optimization (PSO), a kind of meta-heuristic search technique, to pairwise testing (i.e. a special case of combinatorial testing aiming to cover all the pairwise combinations). To systematically build pairwise test suites, we propose two different PSO based algorithms. One algorithm is based on one-test-at-a-time strategy and the other is based on IPO-like strategy. In these two different algorithms, we use PSO to complete the construction of a single test. To successfully apply PSO to cover more uncovered pairwise combinations in this construction process, we provide a detailed description on how to formulate the search space, define the fitness function and set some heuristic settings. To verify the effectiveness of our approach, we implement these algorithms and choose some typical inputs. In our empirical study, we analyze the impact factors of our approach and compare our approach to other well-known approaches. Final empirical results show the effectiveness and efficiency of our approach.

Finding effective approaches to detect vulnerabilities is important to guarantee the security of Web applications. Web application security issues are mostly related to malicious input data and Web forms are the main interface to input these data. According to the above observation, we propose a novel approach to detect Web application vulnerabilities. In our approach, given a URL, we get a target Web form. After analyzing characteristics of this Web form, we assign a set of test values to each field in this form. Then we propose a method to generate test suites taking the weight of each test value into account. Finally, we execute these test suites and analyze corresponding result based on HTTP response code and response HTML. We implement our approach into a tool called D-WAV and choose several Web applications as benchmarks to conduct empirical studies. Final results show that our approach can automatically and effectively discover Web application vulnerabilities such as cross-site scripting and SQL injection.

Interaction testing (also called combinatorial testing) is an cost-effective test generation technique in software testing. Most research work focuses on finding effective approaches to build optimal t-way interaction test suites. However, the strength of different factor sets may not be consistent due to the practical test requirements. To solve this problem, a variable strength combinatorial object and several approaches based on it have been proposed. These approaches include simulated annealing (SA) and greedy algorithms. SA starts with a large randomly generated test suite and then uses a binary search process to find the optimal solution. Although this approach often generates the minimal test suites, it is time consuming. Greedy algorithms avoid this shortcoming but the size of generated test suites is usually not as small as SA. In this paper, we propose a novel approach to generate variable strength interaction test suites (VSITs). In our approach, we adopt a one-test-at-a-time strategy to build final test suites. To generate a single test, we adopt ant colony system (ACS) strategy, an effective variant of ant colony optimization (ACO). In order to successfully adopt ACS, we formulize the solution space, the cost function and several heuristic settings in this framework. We also apply our approach to some typical inputs. Experimental results show the effectiveness of our approach especially compared to greedy algorithms and several existing tools.

Since automated fault localization can improve the efficiency of both the testing and debugging process, it is an important technique for the development of reliable software. This paper proposes a novel fault localization approach based on multi-level similarity of execution traces, which is suitable for object-oriented software. It selects useful test cases at class level and computes code suspiciousness at block level. We develop a tool that implements the approach, and conduct empirical studies to evaluate its effectiveness. The experimental results show that our approach has the potential to be effective in localizing faults for object-oriented software.