The role of reinforcement learning (RL) in enhancing the reasoning of large language models (LLMs) is becoming increasingly significant. Despite the success of RL in many scenarios, there are still many challenges in improving the reasoning of LLMs. One key challenge is the sparse reward, which introduces more training variance in policy optimization and makes it difficult to obtain a good estimation for value function in Actor-Critic (AC) methods. To address these issues, we introduce Direct Advantage-Based Policy Optimization (DAPO), a novel step-level offline RL algorithm with theoretical guarantees for enhancing the reasoning abilities of LLMs. Unlike response-level methods (such as DPO and GRPO) that the update directions of all reasoning steps are governed by the outcome reward uniformly, DAPO employs a critic function to provide step-level dense signals for policy optimization. Additionally, the actor and critic in DAPO are trained independently, ensuring that critic is a good estimation of true state value function and avoiding the co-training instability observed in standard AC methods. We train DAPO on mathematical and code problems and then evaluate its performance on multiple benchmarks. Our results show that DAPO can effectively enhance the mathematical and code capabilities on both SFT models and RL models, demonstrating the effectiveness of DAPO.

Projected policy gradient (PPG) is a basic policy optimization method in reinforcement learning. Given access to exact policy evaluations, previous studies have established the sublinear convergence of PPG for sufficiently small step sizes based on the smoothness and the gradient domination properties of the value function. However, as the step size goes to infinity, PPG reduces to the classic policy iteration method, which suggests the convergence of PPG even for large step sizes. In this paper, we fill this gap and show that PPG admits a sublinear convergence for any constant step sizes. Due to the existence of the state-wise visitation measure in the expression of policy gradient, the existing optimization-based analysis framework for a preconditioned version of PPG (i.e., projected Q-ascent) is not applicable, to the best of our knowledge. Instead, we proceed the proof by computing the state-wise improvement lower bound of PPG based on its inherent structure. In addition, the finite iteration convergence of PPG for any constant step size is further established, which is also new.

Evaluation of an Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) image of the east Gandise, Tibet in China indicates that several important mineralization altered classes can be mapped with good exposure by using ratio image and spectral-angle mapper methods. The resampled ASTER nine-band image data set VNIR+SWIR in 15m spatial resolution was calibrated by using in situ measurements of spectral reflectance. Subsequently, four ASTER ratio images were employed for Al-OH, Fe, Mg-OH, Fe2+, Fe3+ and CO3 alteration extractions. For comparison purpose, in situ surface reflectance data were used to map alteration minerals using spectral angle mapper method for phyllic, propylitic, silicic, argillic, and carbonatic. The result shows that the distribution of the alteration rocks from the spectral angle mapper agrees very well with the alteration zones extracted from the mineralogic ratio images for hydrothermally altered rocks zone associated with copper polymetallic mineralization using ASTER data. The result shows that ASTER data may provide independent parameters useful for regional geological surveys.