Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models (LLMs), especially in scenarios where supervised fine-tuning (SFT) falls short due to limited chain-of-thought (CoT) data.Among RLbased post-training methods, group relative advantage estimation, as exemplified by Group Relative Policy Optimization (GRPO), has attracted considerable attention for eliminating the dependency on the value model, thereby simplifying training compared to traditional approaches like Proximal Policy Optimization (PPO).However, existing group relative advantage estimation method still suffers from training inefficiencies, particularly when the estimated advantage approaches zero.To address this limitation, we propose Advantage-Augmented Policy Optimization (AAPO), a novel RL algorithm that optimizes the crossentropy (CE) loss using advantages enhanced through a margin-based estimation scheme.This approach effectively mitigates the inefficiencies associated with group relative advantage estimation.Experimental results on multiple mathematical reasoning benchmarks and model series demonstrate the superior performance of AAPO.

Neurobiology has made groundbreaking discoveries that have propelled the development of brain-inspired computing. This new field of research has led to the creation of neural pathways that mirror the human brain's visual system, providing numerous transformative insights for modern image processing. Brain-like vision offers remarkable processing efficiency and promises a range of exciting applications, from enhancing video surveillance to improving medical diagnosis. However, current brain-inspired computing models lack portability and efficient power consumption, which limits their widespread use. To address these challenges, we've designed a brain-like color opponent multi-layer neural network model on field programmable gate arrays (FPGAs). This innovative approach enhances portability, reduces power consumption, and increases parallelism in the brain-like computational model. We've conducted rigorous experiments on natural images, and the results are promising. FPGAs can further enhance this innovative approach by improving computational parallelism and enhancing the display of crucial structural information while downplaying trivial details. This integration of biological mechanisms into practical embedded systems is a significant step towards merging these two technologies. Our multi-layer neural network structure aligns with FPGAs' in-memory computing mechanism, bridging the gap between these two technologies. Our work represents a step towards merging them and utilizes physiological color opponent theory to enhance FPGAs' effectiveness in practical image processing applications. By combining the power of neurobiology with the flexibility of FPGAs, we can create more efficient and practical solutions for image processing tasks in various fields.