arXiv 2507.019152025-07-02cs.CLcs.AIcs.LG

梯度自适应策略优化：面向大语言模型多目标对齐

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Lab of AI Safety（人工智能安全国家重点实验室）；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Zhejiang University（浙江大学）

Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing He

备注：: 19 pages, 3 figures. Accepted by ACL 2025 (main)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出梯度自适应策略优化GAPO及其偏好版本P-GAPO，将LLM人类价值对齐建模为多目标优化问题，通过多重梯度下降平衡冲突目标，并在Mistral-7B上验证其在有用性和无害性上优于现有方法。

摘要：

基于人类反馈的强化学习（RLHF）已成为使大语言模型（LLMs）与人类偏好对齐的强大技术。然而，有效地使LLMs与多样化的人类偏好对齐仍然是一个重大挑战，尤其是当这些偏好相互冲突时。为了解决这一问题，我们将人类价值对齐构建为一个多目标优化问题，旨在最大化一组可能相互冲突的目标。我们提出了梯度自适应策略优化（GAPO），这是一种新颖的微调范式，利用多重梯度下降使LLMs与多样化的偏好分布对齐。GAPO自适应地重新缩放每个目标的梯度，以确定一个能在目标之间最优平衡权衡的更新方向。此外，我们引入了P-GAPO，它融合了用户在不同目标上的偏好，并实现更符合用户特定需求的帕累托解。我们的理论分析表明，GAPO会收敛到多目标的帕累托最优解。在Mistral-7B上的实验结果表明，GAPO优于当前最先进的方法，在有用性和无害性方面均取得了更优的性能。

英文摘要：

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict. To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives. We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions. GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the trade-offs between objectives. Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs. Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives. Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。