While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of execution efficiency remains overlooked.We present TRACE, the first benchmark to explicitly assess efficiency in LLM-translated code.TRACE includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency disparities often overlooked by small-scale tests.Using TRACE, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness and efficiency are often misaligned: the correctness leader Claude-Sonnet-4-Think achieves only moderate time efficiency, outperformed by smaller open-source LLMs such as Qwen2.5-Coder-14B-Instruct.2) Inefficiency is both prevalent and patterned: 23.5% of correct translations suffer from notable inefficiency, mainly arising from algorithm implementation discrepancy (11.9%), language construct mismatch (66.4%), and resource management inefficiency (21.7%).3) Inference-time prompt strategies bring only modest improvements, indicating that simple prompting alone is insufficient to improve translation efficiency.Together, our results establish execution efficiency as an essential dimension of code translation and position TRACE as a principled foundation for efficiency-oriented evaluation.Our

Smart contracts have revolutionized the way transactions are executed, offering decentralized and immutable frameworks. The immutability of smart contracts poses significant risks when vulnerabilities exist in their code, leading to financial losses. Despite advancements in using deep learning for smart contract vulnerability detection (SCVD), existing methods struggle with the complex logic and intricate semantics embedded within smart contract code. Large Language Models (LLMs) have shown promise in providing deeper insights into smart contract logic. However, LLMs, such as GPT follow a decoder-only architecture and are trained in an unsupervised manner rather than learning specific labels. In the SCVD task, these LLMs have difficulty in capturing information related to vulnerabilities, leading to very low accuracy. Therefore, we propose CodeXplain, a novel SCVD approach that leverages the deep insights into code from LLM and the supervised learning capabilities of deep learning models to set the latest advance and performance. In particular, we deeply analyze 14 types of dangerous and common smart contract vulnerabilities. Based on the rationale of these vulnerabilities, nine perspective prompts are introduced to guide LLMs in generating code explanations that contribute to SCVD. Then, we propose a CodeT5-based semantic fusion module integrating smart contract code and code explanations. Finally, the performance of SCVD is improved by performing supervised learning on trusted labels. Experimental results on 3,544 real-world smart contracts demonstrate that CodeXplain outperforms 16 state-of-the-art SCVD methods, achieving an F1-score of 94.12% and an accuracy of 93.88%, surpassing all baselines.

The e X tensible M arkup L anguage (XML) is a file format widely used for data transmission in modern software development. In recent years, embedding SQL statements in XML files (i.e., XML-SQL) has become a popular way for developing applications with database access capability. Typically, XML-SQL code snippets demonstrate similar functionalities and structures, leading to repetitive programming work. Therefore, leveraging pre-trained code models for automated code generation presents a promising way to alleviate duplicated efforts and enhance the efficiency of developing XML-SQL code. However, XML-SQL code has strong domain-specific characteristics that general pre-trained code models typically struggle to fully harness, thereby leading to limited overall performance of general pre-trained code models. In this article, we aim to address the challenge of handling this domain-specific knowledge. First, we propose a code updating task and construct the corresponding TwinXSQL dataset to better evaluate the model’s code generation performance in the XML-SQL domain. Then, we leverage the common characteristics of XML-SQL and other programming languages (i.e., all programming languages impose grammar constraints on behavior) to design a bipartite-grammar–aware training framework (named BGA) for unsupervised pre-training, thereby improving the transfer of general-purpose code models to the XML-SQL domain. Specifically, we divide the XML-SQL code into two types of grammatical components: structure components and value components. During pre-training, we undertake three tasks, each designed to learn the internal information of these grammatical components and the relationships between them, enabling the pre-training process to better incorporate previously unlearned domain-specific knowledge of XML-SQL code. Our experimental results show that our trained model XSQLT5-base (220M) improves accuracy by 13.8% compared to the similarly sized CodeT5-base (220M). Additionally, our experiments reveal that ChatGPT, due to its inability to fully learn the XML-SQL domain knowledge, achieves a much lower generation accuracy even with few-shot samples compared to our XSQLT5-base (220M) model.

Modern software development often introduces bug inducing commits (BICs) that can degrade performance or cause crashes. Swift localization of BICs is crucial but challenging due to the entanglement among multiple kinds of information overlooked by existing methods that treat these elements independently. Understanding this entanglement is promising but faces two key challenges: (1) the entanglement representation problem, since simply concatenating diverse data types fails to capture their entanglement effectively; (2) the large input size problem, as identifying BICs requires analyzing a vast set of commits, making simultaneous processing infeasible. To address these challenges, we propose BICSleuth, a framework that encodes the entanglement for effective BIC localization through three stages. First, a small-model-based ranker efficiently narrows down commits despite large input sizes. Second, an LLM-based discriminator deepens the understanding of the entanglement through selective information integration. Third, a reranking strategy combines insights from both stages to enhance localization accuracy. Evaluated on a BIC dataset constructed from Defects4J v2.0.0, BICSleuth outperforms four state-of-the-art approaches, achieving 148.9% of the Mean Reciprocal Rank compared to the best spectrum-based baseline and 507.1% of the MRR of the best IR-based method. Additionally, BICSleuth ranks the BIC first in 70.0% of projects and within the top five in 84.6%. The results demonstrate that BICSleuth effectively leverages the entanglement for BIC localization, with all stages contributing to its success.

Commit messages are important for developers to understand the content and the reason for code changes. However, poor and even empty commit messages widely exist. To improve the quality of commit messages and development efficiency, many commit message generation methods have been proposed. Nevertheless, previous methods mainly focus on a brief generation problem, where both the input code change and the output commit messages are restricted to short. This may initiate a debate on the performance of these methods in practice. In this article, we attempt to remove the restrictions and move the needle forward to a holistic commit message generation problem. In particular, we conduct experiments to evaluate the performance of existing commit message generation methods in holistic commit message generation. In the experiments, we choose seven state-of-the-art commit generation methods and focus on two important scenarios in commit message generation (i.e., the within-project scenario and the cross-project scenario). To conduct our experiments, we publish a holistic commit message dataset HORDA with test data manually labeled. In our evaluations, we find that in generating holistic commit messages, the IR-based method has a better performance than non-pre-trained generation-based methods in the within-project scenario, contradicting previous research findings. Further, while the pre-trained generation-based methods are better than non-pre-trained generation-based methods, they are still constrained by the limitations of generation models.

Prevalent Fault Localization (FL) techniques rely on tests to localize buggy program elements. Tests could be treated as fuel to further boost FL by providing more debugging information. Therefore, it is highly valuable to measure the Fault Diagnosis Capability (FDC) of a test for diagnosing faults, so as to select or generate tests to better help FL (i.e., FL-oriented test selection or FL-oriented test generation). To this end, researchers have proposed many FDC metrics, which serve as the selection criterion in FL-oriented test selection or the fitness function in FL-oriented test generation. Existing FDC metrics can be classified into result-agnostic and result-aware metrics depending on whether they take test results (i.e., passing or failing) as input. Although result-aware metrics perform better in test selection, they have restricted applications due to the input of test results, e.g., they cannot be applied to guide test generation. Moreover, all the existing FDC metrics are designed based on some pre-defined heuristics and have achieved limited FL performance due to their inaccuracy. To address these issues, in this article, we reconsider result-agnostic metrics (i.e., metrics that do not take test results as input), and propose a novel result-agnostic metric RLFDC which predicts FDC values of tests through reinforcement learning. In particular, we treat FL results as reward signals, and train an FDC prediction model with the direct FL feedback to automatically learn a more accurate measurement rather than design one based on pre-defined heuristics. Finally, we evaluate the proposed RLFDC on Defects4J by applying the studied metrics to test selection and generation. According to the experimental results, the proposed RLFDC outperforms all the result-agnostic metrics in both test selection and generation, e.g., when applied to selecting human-written tests, RLFDC achieves 28.2% and 21.6% higher acc@1 and mAP values compared to the state-of-the-art result-agnostic metric TfD. Besides, RLFDC even achieves competitive performance compared to the state-of-the-art result-aware metric FDG in test selection.

Code pre-trained models have shown promising effectiveness in various software engineering tasks. Among these tasks, many tasks are related to software evolution and/or code editing. However, existing code pre-trained models often overlook the real-world code editing data and the evolutionary nature of the editing process. In this paper, to simulate the step-by-step code editing process of human developers, we propose DivoT5, a pre-trained model based on directional diffusion at the data level. In DivoT5, we adopt two categories of pre-training tasks. The first category is mask and denoising tasks augmented with a diffusion direction representing code evolution. That is, we first apply a noising process to the code snippets before evolution, and then ask the pre-training process to restore the snippets with noise into the code snippets after evolution. The second category is tasks aiming to reinforce the evolutionary direction. That is, we first generate various intermediate versions for each pair of snippets before and after evolution, and then ask the pre-training process to transform the intermediate versions into the snippet after evolution for each pair. We evaluate DivoT5 for two code-editing scenarios (including a number of tasks) and one non-editing scenario using four downstream tasks. For each downstream task, we fine-tune the pre-trained DivoT5 on multiple corresponding datasets and evaluate its effectiveness across diverse scenarios Our experimental results show that ivoT5 achieves state-of-the-art (SOTA) performance on most tasks in comparison to models of the same scale (220M), large-scale (770M, 6.7B) models in fine-tuning, and billion-scale (6.7B, 8B, ChatGPT) instruct models in few-shot settings. For one code-editing task (i.e., CodeReview in NL-based CodeRefinement task), DivoT5 pre-trained on top of CodeT5-small (60M) can even outperform CodeT5-base (220M) and other pre-trained models with 220M parameters except for DivoT5 pre-trained on top of CodeT5-base (220M).

LLMs demonstrate significant potential across various software engineering tasks. However, they still face challenges in generating correct code on the first attempt when addressing complex requirements. Introducing a discriminator to select reliable outputs from multiple generated results is an effective way to enhance their reliability and stability. Currently, these discriminators fall into two categories: execution-based discriminators and non-execution-based discriminators. Execution-based discriminators face flexibility challenges due to difficulties in obtaining test cases and security concerns, while non-execution-based discriminators, although more flexible, struggle to capture subtle differences in code details. To maintain flexibility while improving the model’s ability to capture fine-grained code details, this paper proposes Condor. We first design contrastive learning to optimize the code representations of the base model, enabling it to reflect differences in code details. Then, we leverage intermediate data from the code modification process to further enrich the discriminator’s training data, enhancing its ability to discern code details. Experimental results indicate that on the subtle code difference dataset (i.e., CodeNanoFix), Condor significantly outperforms other discriminators in discriminative performance: Condor (1.3B) improves the discriminative F1 score of DeepSeek-Coder (1.3B) from 67% to 73%. In discriminating LLM-generated outputs, Condor (1.3B) and Condor (110M) raise the Pass@1 score of Llama-3.1-Instruct (70B) on the CodeNanoFix dataset from 52.64% to 62.63% and 59.64%, respectively. Moreover, Condor demonstrates strong generalization capabilities on the APPS, MBPP, and LiveCodeBench datasets. For example, Condor (1.3B) improves the Pass@1 of Llama-3.1-Instruct (70B) on the APPS dataset by 147.05%.

Code generation has gained increasing attention as a task to automate software development by transforming high-level descriptions into executable code. While large language models (LLMs) are effective in generating code, their performance heavily relies on the quality of input prompts. Current prompt engineering methods involve manual effort in designing prompts, which can be time-consuming and yield inconsistent results, potentially constraining the efficacy of LLMs in practical applications. This paper introduces Prochemy, a novel approach for automatically refining prompts iteratively to enhance code generation. Prochemy addresses the limitations of manual prompt engineering by automating the optimization process, ensuring prompt consistency during inference, and aligning with multi-agent systems. It iteratively refines prompts based on model performance, using an optimized final prompt to improve consistency and reliability across tasks. We evaluate Prochemy on both natural language-based code generation and code translation tasks using three series of LLMs. Results show that when combining Prochemy with existing approaches, it outperforms baseline prompting methods. It achieves improvements of 5.0% (GPT-3.5-Turbo) and 1.9% (GPT-4o) over zero-shot baselines on HumanEval. For the state-of-the-art LDB, Prochemy + LDB outperforms standalone methods by 1.2–1.8%. For code translation, Prochemy elevates GPT-4o’s performance on Java-to-Python (AVATAR) from 74.5 to 84.1 (+12.9%) and Python-to-Java from 66.8 to 78.2 (+17.1%). Furthermore, considering that the o1-mini model integrates prompt engineering techniques, Prochemy can continue to show good performance among it, further validating its effectiveness in code generation and translation tasks. Additionally, Prochemy is designed to be plug-and-play, optimizing prompts with minimal human intervention and seamlessly bridging the gap between simple prompts and complex frameworks.

Pretrained models for code have exhibited promising performance across various code-related tasks, such as code summarization, code completion, code translation, and bug detection. However, despite their success, the majority of current models still represent code as a token sequence, which may not adequately capture the essence of the underlying code structure.

In Continuous Integration (CI), accurate build prediction is crucial for minimizing development costs and enhancing efficiency. However, existing build prediction methods, typically based on predefined rules or machine learning classifiers employing feature engineering, have been constrained by their limited ability to fully capture the intricate details of commit artifacts, such as code change and commit messages. These artifacts are critical for understanding the commit under a build but have been inadequately utilized in existing approaches. To address this problem, we propose GitSense, a Transformer-based model specifically designed to incorporate the rich and complex information contained within commit artifacts for the first. GitSense employs an advanced textual encoder with built-in sliding window text samplers for textual features and a statistical feature encoder for extracted statistical features. This innovative approach allows for a comprehensive analysis of lengthy and intricate commit artifacts, surpassing the capabilities of traditional methods. We conduct comprehensive experiments to compare GitSense with five state-of-the-art build prediction models, Longformer, and ChatGPT. The experimental results show that GitSense outperforms these models in predicting failed builds, evidenced by 32.7%-872.1.0% better on F1-score, 23.9%-437.5% better on Precision, and 40.2%-1396.0% better on Recall.

Mutation testing is widely used to measure the test adequacy of a project. Despite its popularity, mutation testing is time-consuming and extremely expensive. To mitigate this problem, researchers propose Predictive Mutation Testing (PMT). Existing PMT approaches build classification models based on statistical program features or source code of programs to predict mutation testing results. Previous statistical feature-based PMT models need expensive overhead to collect dynamic features and neglect the rich information inherent in code text. Previous text-based PMT models extract essential code elements as input and outperform the feature-based models. However, they encode code text in a plain way. Therefore, they cannot sensitively capture subtle differences in mutants and they have difficulty in capturing the correlation between mutants and tests. To address these challenges, we propose a new model, SODA. SODA uses a new learning strategy, Mutational Semantic Learning, to make our model spot code mutation and its impact on test behavior. In particular, we employ a new sampling strategy to reinforce the corresponding relationship between mutants and tests by sampling same-mutant contrastive groups. Then we employ contrastive learning to make our model capture subtle differences in mutants. We conduct experiments to investigate the performance of SODA. The results demonstrate that both in the cross-project and cross-version scenarios, SODA achieves state-of-the-art classification performance (improves upon baselines by 5.32%-114.92% in kill-F1 score, 0.04%-25.54% in survive-F1 score, 4.25%-60.43% in accuracy) and has the lowest mutation score error.

Recently, neural techniques have been used to generate source code automatically. While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages. Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts. In this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language. To our knowledge, this is the first turducken-style code generation task. For this task, we present Lyra: a dataset in Python with embedded SQL. This dataset contains 2,000 carefully annotated database manipulation programs from real usage projects. Each program is paired with both a Chinese comment and an English comment. In our experiment, we adopted Transformer, BERT-style, and GPT-style models as baselines. In the best setting, GPT-style model can achieve 24% and 25.5% AST exact matching accuracy using Chinese and English comments, respectively. Therefore, we believe that Lyra provides a new challenge for code generation. Yet, overcoming this challenge may significantly boost the applicability of code generation techniques for real-world software development.