Enhancing the mathematical reasoning capabilities of Large Language Models (LLMs) is of great scientific and practical significance.Researchers typically employ process-supervised reward models (PRMs) to guide the reasoning process, effectively improving the models' reasoning abilities.However, existing methods for constructing process supervision training data, such as manual annotation and perstep Monte Carlo estimation, are often costly or suffer from poor quality.To address these challenges, this paper introduces a framework called EpicPRM (Efficient, Precise, Cheap), which annotates each intermediate reasoning step based on its quantified contribution and uses an adaptive binary search algorithm to enhance both annotation precision and efficiency.Using this approach, we efficiently construct a high-quality process supervision training dataset named Epic50k, consisting of 50k annotated intermediate steps.Compared to other publicly available datasets, the PRM trained on Epic50k demonstrates significantly superior performance.Getting Epic50k at

Recent advances have demonstrated that integrating reinforcement learning with rule-based rewards can significantly enhance the reasoning capabilities of large language models (LLMs), even without supervised fine-tuning (SFT). However, prevalent reinforcement learning algorithms such as GRPO and its variants like DAPO, suffer from a coarse granularity issue when computing the advantage. Specifically, they compute rollout-level advantages that assign identical values to every token within a sequence, failing to capture token-specific contributions. To address this limitation, we propose Key-token Advantage Estimation (KTAE)—a novel algorithm that estimates fine-grained, token-level advantages without introducing additional models. KTAE leverages the correctness of sampled rollouts and applies statistical analysis to quantify the importance of individual tokens within a sequence to the final outcome. This quantified token-level importance is then combined with the rollout-level advantage to obtain a more fine-grained token-level advantage estimation. Empirical results show that models trained with GRPO+KTAE and DAPO+KTAE outperform baseline methods across five mathematical reasoning benchmarks. Notably, they achieve higher accuracy with shorter responses and even surpass R1-Distill-Qwen-1.5B using the same base model.

Instruction tuning is a vital step of training large language models (LLMs), so how to enhance the effect of instruction tuning has received increased attention. Existing works indicate that the quality of the dataset is more crucial than the quantity during instruction tuning of LLMs. Therefore, recently a lot of studies focus on exploring the methods of selecting high-quality subset from instruction datasets, aiming to reduce training costs and enhance the instruction-following capabilities of LLMs. This paper presents a comprehensive survey on data selection for LLM instruction tuning. Firstly, we introduce the wildly used instruction datasets. Then, we propose a new taxonomy of the data selection methods and provide a detailed introduction of recent advances, and the evaluation strategies and results of data selection methods are also elaborated in detail. Finally, we emphasize the open challenges and present new frontiers of this task.

This article proposes to conduct natural language inference with novel Enhanced-Relation-Head-Dependent triplets (RHD triplets) , which are constructed via enhancing each word in the RHD triplet with its associated local context. Most previous approaches based on deep neural network (DNN) for this task either perform token alignment without considering syntactic dependency among words, or directly use tree- LSTM to generate passage representation with irrelevant information. To improve token alignment and inference judgment with word-pair-dependency, the RHD triplet structure is first proposed. To avoid incorporating irrelevant information, this proposed approach performs comparison directly on each triplet-pair of the given passage-pair (instead of comparing each triplet in a passage with the content merged from the whole opposite passage). Furthermore, to take local context into consideration while conducting token alignment and inference judgment, we also enhance the words of the triplets with their associated local context to improve the performance. Experimental results show that the proposed approach is better than most previous approaches that adopt tree structures, and its performance is comparable to other state-of-the-art approaches (however, our approach is more human comprehensible).

This paper proposes to perform natural language inference with Word-Pair-Dependency-Triplets. Most previous DNN-based approaches either ignore syntactic dependency among words, or directly use tree-LSTM to generate sentence representation with irrelevant information. To overcome the problems mentioned above, we adopt Word-Pair-Dependency-Triplets to improve alignment and inference judgment. To be specific, instead of comparing each triplet from one passage with the merged information of another passage, we first propose to perform comparison directly between the triplets of the given passage-pair to make the judgement more interpretable. Experimental results show that the performance of our approach is better than most of the approaches that use tree structures, and is comparable to other state-of-the-art approaches.