Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences.However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict.To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives.We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions.GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the tradeoffs between objectives.Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs.Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives.Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.

Nowadays, most high-precision products are still assembled manually, which leads to a low one-time pass rate of products. Workers need to adjust unqualified products repeatedly based on experience, resulting in inefficiency and poor quality consistency. In this work, we propose an explore-exploit reinforcement learning (EERL) framework to suggest the assembly parameters and quantity that workers need to adjust at each step. Jointed with the pretrained product performance prediction model, EERL can output a sequential decision to guide workers hand in hand to adjust unqualified products. EERL includes a two-phase learning process: 1) exploration; and 2) exploitation. In exploration phase, agents are encouraged by the curiosity to fully explore the qualified assembly states in the assembly-adjustment feature space. The regulated difference of random network distillation is used as a measure of curiosity. During exploitation, the agent is trained to learn an assembly-adjustment guidance policy of moving from any unqualified initial assembly state to the corresponding qualified state while satisfying the assembly-adjustment constraints. The curriculum learning mechanism is introduced to learn effectively in the complex environment with sparse reward and adjustment constraints. The proposed approach is validated on an benchmark optimization function and a case study of the gyroscope. The experimental results demonstrate that the proposed approach outperforms other existing approaches.