arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.35480cs.LGcs.CL
收藏

GPDPO:通过群体帕累托优势策略优化从奖励冲突中学习

GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

Sun Yat-sen University(中山大学);University of Trento(特伦托大学)

Zixuan Xu, Haoxuan Chen, Jiangxin Sun, Jian-Fang Hu

AI总结:

针对多奖励强化学习中奖励冲突导致的优化问题,提出GPDPO方法,通过识别采样响应的帕累托优势关系转换为辅助偏好信号,在API-Bank实验中优于基线,可减少奖励冲突。

摘要:

多奖励强化学习为将大语言模型(LLMs)与多样化目标对齐提供了有前景的框架,但冲突的奖励信号会引发模糊的更新方向,阻碍稳定优化。现有方法通过奖励重加权、梯度协调或 rollout 过滤来解决此类冲突,但它们主要基于单响应或群体级优化信号操作,且未充分利用采样响应间的关系信息。因此,经冲突过滤的 rollout 可能仍包含训练过程中被丢弃的有用监督信息。我们提出群体帕累托优势策略优化(GPDPO),这是一种多奖励策略优化方法,可从经冲突过滤的 rollout 中恢复成对监督信息。GPDPO 识别采样响应间的帕累托优势关系,并将其转换为辅助偏好信号,与原始策略目标联合优化。这使得在无需额外采样或奖励评估的情况下,更有效地复用现有 rollout。在 API-Bank 上针对多种模型规模和奖励配置开展的实验表明,GPDPO 始终优于具有竞争力的多奖励优化基线。进一步分析显示,GPDPO 可减少群体相对和成对奖励冲突,证明利用关系监督进行多奖励对齐的有效性。代码将发布。

英文摘要:

Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑