SeRA：使用隐式奖励边际对大型语言模型进行自我审查与对齐
SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
AI总结：
提出SeRA方法，通过隐式奖励边际进行样本选择和偏好引导，解决直接对齐算法中离线策略数据导致的虚假相关和过拟合问题，有效提升LLMs训练。
摘要：
直接对齐算法（DAAs），如直接偏好优化（DPO），因其简单性、高效性和稳定性，已成为基于人类反馈的强化学习（RLHF）的流行替代方案。然而，DAAs中使用的偏好通常在对齐训练开始前收集且保持不变（离线策略）。这可能导致两个问题：策略模型（1）捕捉到数据集中的虚假相关性（而非学习人类偏好标签中表达的对齐意图），以及（2）过拟合离线策略轨迹上的反馈，而这些轨迹被更新后的策略模型生成的可能性较低。为解决这些问题，我们引入了自我审查与对齐（SeRA），这是一种低成本且有效的方法，可与现有DAAs轻松结合。SeRA包含两个组件：（1）使用隐式奖励边际进行样本选择，有助于缓解对某些不良特征的过拟合，以及（2）使用隐式奖励进行偏好引导，以低成本方式用更新后的策略模型增强偏好数据。包括指令遵循任务在内的广泛实验，证明了SeRA在用DAAs离线偏好数据集训练LLMs时的有效性和通用性。
英文摘要：
Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment training begins and remain unchanged (off-policy). This can lead to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by an updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
