arXiv 2310.002122023-10-10cs.LGcs.AIcs.CL

成对近端策略优化：利用相对反馈进行大语言模型对齐

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

University of California, Berkeley（加州大学伯克利分校）

Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, Jiantao Jiao

备注：: 19 pages, 5 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出成对近端策略优化（P3O），一种基于相对反馈的轨迹级策略梯度算法，用于大语言模型对齐，理论上对等价奖励不变，实证优于PPO，实现更简单有效的对齐。

摘要：

大型语言模型（LLMs）可以通过在大规模语料库上进行预训练来获取广泛的世界知识。然而，由于接触到低质量数据，LLMs可能会表现出有害行为，而未能与人类价值观对齐。引导LLMs走向有益行为的主流方法涉及基于人类反馈的强化学习（RLHF），其中近端策略优化（PPO）作为默认的RL优化器。尽管PPO有效，但在优化基于比较损失训练出的奖励时存在局限性。首先，由于需要校准奖励尺度，PPO对于包含相同偏好信息的等价奖励函数并非不变。其次，与轨迹级优化相比，PPO需要逐词更新，这增加了函数逼近和算法设计的复杂性。本文提出了一种新框架——基于相对反馈的强化学习，以及一种新颖的轨迹级策略梯度算法——成对近端策略优化（P3O），该算法直接作用于比较奖励。我们从理论上证明，P3O对等价奖励具有不变性，并避免了PPO的复杂性。实证评估表明，P3O在KL-奖励权衡方面优于PPO，并且能够与人类偏好对齐，效果不逊于甚至优于先前的方法。总之，这项工作通过相对反馈引入了一种更简单但有效的LLM与人类偏好对齐方法。

英文摘要：

Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as the default RL optimizer. Despite its effectiveness, PPO has limitations when optimizing rewards trained from comparison-based loss. Primarily, PPO is not invariant to equivalent reward functions containing identical preference information due to the need to calibrate the reward scale. Additionally, PPO's necessity for token-wise updates introduces complexity in both function approximation and algorithm design compared to trajectory-wise optimization. This paper proposes a new framework, reinforcement learning with relative feedback, and a novel trajectory-wise policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O) that operates directly on comparative rewards. We show theoretically that P3O is invariant to equivalent rewards and avoids the complexity of PPO. Empirical evaluations demonstrate that P3O outperforms PPO in the KL-Reward trade-off and can align with human preferences as well as or better than prior methods. In summary, this work introduces a simpler yet effective approach for aligning LLMs to human preferences through relative feedback.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。