Modern Large Language Model (LLM)-based programming agents often rely on test execution feedback to refine their generated code. These tests are synthetically generated by LLMs. However, LLMs may produce invalid or hallucinated test cases, which can mislead feedback loops and degrade the performance of agents in refining and improving code. This paper introducesVALTEST, a novel framework that leverages semantic entropy to automatically validate test cases generated by LLMs. By analyzing the semantic structure of test cases and computing entropy-based uncertainty measures,VALTESTtrains a machine learning model to classify test cases as valid or invalid and filters out invalid test cases. Experiments on multiple benchmark datasets and various LLMs show thatVALTESTnot only boosts test validity by up to 29% but also improves code generation performance, as evidenced by significant increases in pass@1 scores. Our extensive experiments also reveal that semantic entropy is a reliable indicator to distinguish between valid and invalid test cases and provides a robust solution for improving the correctness of LLM-generated test cases used in software testing and code generation.

Ensuring correctness is crucial for code generation. Formal verification offers a definitive assurance of correctness, but demands substantial human effort in proof construction and hence raises a pressing need for automation. The primary obsta- cle lies in the severe lack of data—there is much fewer proofs than code snippets for Large Language Models (LLMs) to train upon. In this paper, we introduce SAFE, a framework that overcomes the lack of human-written proofs to enable automated proof generation of Rust code. SAFE establishes a self-evolving cycle where data synthesis and fine-tuning collaborate to enhance the model capability, leveraging the definitive power of a symbolic verifier in telling correct proofs from incorrect ones. SAFE also re-purposes the large number of synthesized incorrect proofs to train the self-debugging capability of the fine-tuned models, empowering them to fix incorrect proofs based on the verifier’s feedback. SAFE demonstrates superior efficiency and precision compared to GPT-4o. Through tens of thousands of synthesized proofs and the self-debugging mechanism, we improve the capa- bility of open-source models, initially unacquainted with formal verification, to automatically write proofs for Rust code. This advancement leads to a signifi- cant improvement in performance, achieving a 52.52% accuracy rate in a bench- mark crafted by human experts, a significant leap over GPT-4o’s performance of 14.39%.

Traditionally, energy efficiency research has focused on reducing energy consumption at the hardware level and, more recently, in the design and coding phases of the software development life cycle. However, software testing’s impact on energy consumption did not receive attention from the research community. Specifically, how test code design quality and test smell (e.g., sub-optimal design and bad practices in test code) impact energy consumption has not been investigated yet. This study aims to examine open-source software projects to analyze the association between test smell and its effects on energy consumption in software testing. We conducted a mixed-method empirical analysis from two perspectives; software (data mining in 12 Apache projects) and developers’ views (a survey of 62 software practitioners). Our findings show that: (1) test smell is associated with energy consumption in software testing. Specifically, the smelly part of a test case consumes more energy compared to the non-smelly part. (2) certain test smells are more energy-hungry than others, (3) refactored test cases tend to consume less energy than their smelly counterparts, and (4) most developers (45 % of the survey respondents) lack knowledge about test smells’ impact on energy consumption. Based on the results, we emphasize raising developers awareness regarding the impact of test smells on energy consumption. Additionally we present several observations that can direct future research and developments.

Generative AI has shown its value for many software engineering tasks. Still in its infancy, large language model (LLM)-based proof generation lags behind LLM-based code generation. In this paper, we present A uto V erus . A uto V erus uses LLMs to automatically generate correctness proof for Rust code. A uto V erus is designed to match the unique features of Verus, a verification tool that can prove the correctness of Rust code using proofs and specifications also written in Rust. A uto V erus consists of a network of agents that are crafted and orchestrated to mimic human experts’ three phases of proof construction: preliminary proof generation, proof refinement guided by generic tips, and proof debugging guided by verification errors. To thoroughly evaluate A uto V erus and help foster future research in this direction, we have built a benchmark suite of 150 non-trivial proof tasks, based on existing code-generation benchmarks and verification benchmarks. Our evaluation shows that A uto V erus can automatically generate correct proof for more than 90% of them, with more than half of them tackled in less than 30 seconds or 3 LLM calls.

Researchers and tool developers working on dynamic analysis, software testing, automated program repair, verification, and validation, need large compiled, compilable, and executable code corpora to test their ideas. The publicly available corpora are relatively small, and/or non-compilable, and/or non-executable. Developing a compiled code corpus is a laborious activity demanding significant manual effort and human intervention. To facilitate large-scale program analysis research, we develop SourcererJBF , a J ava B uild F ramework that can automatically build a large Java code corpus without project-specific instructions and human intervention. To generate a compiled code corpus, SourcererJBF creates an offline knowledge base by collecting external dependencies from the project directories and existing build scripts (if available). It constructs indices of those collected external dependencies that enable a fast search for resolving dependencies during the project compilation. As the output of the large-scale compilation, it produces JAigantic, a compilable Java corpus containing compiled projects, their bytecode, dependencies, normalized build script, and build command. We evaluated SourcererJBF’s effectiveness, correctness, performance, and scalability in a large collection of Java projects. Our experimental results demonstrate that SourcererJBF is significantly effective and scalable in building large Java code corpus. Besides, it substantiates reasonable performance and correctness similar to projects’ existing build systems.

Online code clones occur due to reusing code snippets in software repositories from online resources such as GitHub and Stack Overflow. Previous works have shown that snippets from Stack Overflow are reused in other open-source projects and vice versa. Analysis of online code reusing patterns could identify outdated code, understand developers' practices, and help to design new code search engines. This study analyzed JavaScript online code clones between Stack Overflow and GitHub repositories. We first developed a JavaScript code corpus to search online clones. The clone search results reported 12,579 online clones between 276,547 non-trivial syntactically validated Stack Overflow snippets and 292 GitHub repositories. We manually classified the top 10% (1257) pairs of clones in seven online clone patterns. We observed that around 70% of JavaScript snippets in Stack Overflow posts are copied from GitHub repositories or from other external sources. Moreover, only 30.59% of JavaScript Snippets in Stack Overflow accepted answers could be considered as reusable snippets.

Code cloning is a common code reusing technique that occurs when developers replicate similar pieces of code fragments within or between software repositories. Another replication happens when developers repeat method interfaces (i.e., method name, return and parameter types). Two methods are prone to be cloned when those have similar interfaces and perform similar functionalities. Considering this, a new lightweight Interface Driven Code Clone Detection (IDCCD) technique is proposed, that can detect clones by using method interface similarities. First, the method blocks are tokenized from the source files. For those method block tokens, interface information is extracted and indexed with mapped tokens. Then, similar interfaces are queried from that index and compared those with a similarity function for detecting clones. IDCCD is evaluated with other state of the art techniques by using BigCloneEval framework. The experimental results show that IDCCD performs similar comparing to other existing tools with a lower complexity.