Sijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang, Ke Wang, Jinbo Su, Mengshu Sun, Lei Liang, Jing Zhang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Current large language models (LLMs) often struggle to produce accurate solutions on the first attempt for code generation.Prior research tackles this challenge by generating multiple candidate solutions and validating them with LLM-generated unit tests.The execution results of unit tests serve as reward signals to identify correct solutions.As LLMs always confidently make mistakes, these unit tests are not reliable, thereby diminishing the quality of reward signals.Motivated by the observation that scaling the number of solutions improves LLM performance, we explore the impact of scaling unit tests to enhance reward signal quality.Our pioneer experiment reveals a positive correlation between the number of unit tests and reward signal quality, with greater benefits observed in more challenging problems.Based on these insights, we propose CodeRM-8B, a lightweight yet effective unit test generator that enables efficient and high-quality unit test scaling.Additionally, we implement a dynamic scaling mechanism that adapts the number of unit tests based on problem difficulty, further improving efficiency.Experimental results show that our approach significantly improves performance across various models on three benchmarks (e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus).

Current inference scaling methods, such as Self-consistency and Best-of-N, have proven effective in improving the accuracy of LLMs on complex reasoning tasks.However, these methods rely heavily on the quality of candidate responses and are unable to produce correct answers when all candidates are incorrect.In this paper, we propose a novel inference scaling strategy, CoT-based Synthesizer, which leverages CoT reasoning to synthesize superior answers by analyzing complementary information from multiple candidate responses, even when all candidate responses are flawed.To enable a lightweight and cost-effective implementation, we introduce an automated data generation pipeline that creates diverse training data.This allows smaller LLMs trained on this data to improve the inference accuracy of larger models, including API-based LLMs.Experimental results across four benchmark datasets with seven policy models demonstrate that our method significantly enhances performance, with gains of 11.8% for Llama3-8B and 10.3% for GPT-4o on the MATH500 dataset.The corresponding training data and code are publicly available on the repository 1 .

We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values.Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty.

This study aims to understand the self-regulation strategies first-year engineering students use to cope with emotions during programming tasks. We used Zimmerman's framework to identify the processes of self- regulated learning (SRL) as students worked on programming tasks [1, 2]. The SRL framework is a cyclical process that involves three main stages: forethought (preparation for the task), performance (engagement with the task), and self-reflection (reflection on their performance on the task). Most literature about SRL focuses on how students regulate their learning during the forethought and self-reflection stages [3, 4]. There is very little attention on students’ self-regulated learning experiences during the performance stage because it is hard to observe students while they work on the task. This study provides a unique opportunity to understand students’ self-regulation as they worked on programming problems. Seventeen first-year engineering students at a large midwestern university in the United States participated in this study during Spring 2018 [5]. As students worked on the programming task, multi-modal data were collected (video screen capture, eye-gaze data, facial expressions). Following the programming task, students reflected on their experience in a retrospective think-aloud interview.