The capability of the reward model (RM) is crucial for the success of Reinforcement Learning from Human Feedback (RLHF) in aligning with human preferences. However, as training progresses, the output space distribution of the policy model shifts. The RM, initially trained on responses sampled from the output distribution of the early policy model, gradually loses its ability to distinguish between responses from the newly shifted distribution. This issue is further compounded when the RM, trained on a specific data distribution, struggles to generalize to examples outside of that distribution. These two issues can be united as a challenge posed by the shifted distribution of the environment. To surmount this challenge, we introduce MetaRM, a novel method leveraging meta-learning to adapt the RM to the shifted environment distribution. MetaRM optimizes the RM in an alternating way, by preserving both the preferences of the original preference pairs, as well as maximizing discrimination power over new examples of the shifted distribution. Extensive experiments demonstrate that MetaRM can iteratively enhance the performance of human preference alignment by improving the RM's capacity to identify subtle differences in samples of shifted distributions.

The growing use of large language models (LLMs) has increased the importance of natural language (NL) in software engineering. However, ambiguity of NL can harm software quality, as unclear problem descriptions may lead to incorrect program generation. Detecting and resolving such ambiguity is challenging, motivating our introduction of the automated repair of ambiguous NL descriptions, which we approach by reducing code generation uncertainty and better aligning NL with input–output examples. Ambiguity repair is difficult for LLMs because they must understand how their interpretation of a description changes when the text is altered. We find that directly prompting LLMs to clarify ambiguity often produces irrelevant or inconsistent edits. To address this, we decompose this task into two simpler steps: (1) analyzing and repairing the LLM’s interpretation of the description — captured by the distribution of programs it induces — using traditional testing and program repair, and (2) refining the description based on distribution changes via a method we call contrastive specification inference. We implement this approach in a tool called SPEC-FIX and evaluate it using four state-of-the-art LLMs (GPT-4o, GPT-4o-mini, DeepSeek-V3, and Qwen2.5-Coder-32B-Instruct) on three popular code generation benchmarks (HumanEval+, MBPP+ and LiveCodeBench). Without human intervention or external information, SPECFIX modified 43.58% of descriptions, improving Pass@1 on the modified set by 30.9%. This yields a 4.09% absolute improvement across the entire benchmark. Repairs also transfer across models: descriptions repaired for one model improve other models’ performance by 10.48%.

Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Software systems often encounter various errors or exceptions in practice, and thus proper error handling code is essential to ensure the reliability of software systems. Unfortunately, error handling code is often bug-prone, while sufficiently testing them is challenging as such code often cannot be triggered under normal conditions. Motivated by this, recent studies have proposed to leverage software fault injection (SFI) based fuzzing to discover potential bugs in complicated error handling code. Despite the promising results achieved, their effectiveness and efficiency are still compromised in practice due to the huge search space of error sites, inadequate fuzzing guidance, and the overhead induced by context-sensitive SFI. To achieve effective and efficient testing of error handling code, this study presents AFL-FI, which first utilizes a similarity-based method to identify suspicious error sites, and then incorporates the idea of error site coverage to guide the fuzzing process. Finally, the design of lightweight context-sensitive SFI enables AFL-FI to execute test cases efficiently. We evaluate AFL-FI on eight large-scale open-source projects, and the results show that it can outperform existing state-of-the-art fuzzing tools significantly in terms of branch code coverage. More importantly, AFL-FI has discovered 13 previously unknown bugs, and all of them have been confirmed while 12 of them have been fixed. Besides, our evaluation also demonstrates that all the key designs of AFL- F I are effective that contribute significantly to its overall performance.

Java Virtual Machine (JVM) is the fundamental software system that supports the interpretation and execution of Java bytecode. To support the surging performance demands for the increasingly complex and large-scale Java programs, Just-In-Time (JIT) compiler was proposed to perform sophisticated runtime optimization. However, this inevitably induces various bugs, which are becoming more pervasive over the decades and can often cause significant consequences. To facilitate the design of effective and efficient testing techniques to detect JIT compiler bugs. This study first performs a preliminary study aiming to understand the characteristics of JIT compiler bugs and the corresponding triggering test cases. Inspired by the empirical findings, we propose JOpFuzzer, a new JVM testing approach with a specific focus on JIT compiler bugs. The main novelty of JOpFuzzer is embodied in three aspects. First, besides generating new seeds, JOpFuzzer also searches for diverse configurations along the new dimension of optimization options. Second, JOpFuzzer learns the correlations between various code features and different optimization options to guide the process of seed mutation and option exploration. Third, it leverages the profile data, which can reveal the program execution information, to guide the fuzzing process. Such nov-elties enable JOpFuzzer to effectively and efficiently explore the two-dimensional input spaces. Extensive evaluation shows that JOpFuzzer outperforms the state-of-the-art approaches in terms of the achieved code coverages. More importantly, it has detected 41 bugs in OpenJDK, and 25 of them have already been confirmed or fixed by the corresponding developers.

Third-party libraries (TPLs) are widely utilized by Android developers to implement new apps. Unfortunately, TPLs are often suffering from various vulnerabilities, which could be exploited by attackers to cause catastrophic consequences for app users. Therefore, testing whether a vulnerability has been patched in target apps is crucial. However, existing techniques are unable to effectively test patch presence for obfuscated apps while obfuscation is pervasive in practice. To address the new challenges introduced by code obfuscation, this study presents PHunter, which is a system that captures obfuscation-resilient semantic features of patch-related methods to identify the presence of the patch in target apps. Specifically, PHunter utilizes coarse-grained features to locate patch-related methods, and compares the fine-grained semantic similarity to determine whether the code has been patched. Extensive evaluations on 94 CVEs and 200 apps show that PHunter can outperform state-of-the-art tools, achieving an average accuracy of 97.1% with high efficiency and low false positive rates. Besides, PHunter is able to be resilient to different obfuscation strategies. More importantly, PHunter is useful in eliminating the false alarms generated by existing TPL detection tools. In particular, it can help reduce up to 25.2% of the false alarms with an accuracy of 95.3%.

SMT solvers are utilized to check the satisfiability of logic formulas and have been applied in various crucial domains, including software verification, test case generation, and program synthesis. However, bugs hidden in SMT solvers can lead to severe consequences, causing erroneous results in these domains. Therefore, ensuring the reliability and robustness of SMT solvers is of critical importance. Despite several testing approaches proposed for SMT solvers, generating effective test formulas to comprehensively test SMT solvers remains a challenge. To address this challenge, in this study, we propose to port large language models (LLMs) to generate SMT formulas for fuzzing solvers. Specifically, the study presents a novel retrain-finetune pipeline to unleash the potential of language models to generate effective SMT formulas and improve their generation performance through data augmentation. We implemented our approach as a practical fuzzing tool, named LasT,and then extensively tested the state-of-the-art SMT solvers, namely Z3, cvc5, and Bitwuzla. To date, Last has successfully uncovered 65 genuine bugs for the solvers, of which 45 have been fixed by the developers.