通过偏好重排序与基于表征的奖励建模实现大语言模型的高效安全对齐
Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
AI总结：
针对大语言模型安全对齐中强化学习算法的分布偏移问题，提出基于偏好重排序与表征奖励建模的框架，利用模型内在安全判断提取奖励信号实现偏好重排，在提升安全性能的同时大幅降低计算开销。
摘要：
用于大语言模型（LLMs）安全对齐的强化学习（RL）算法，如直接偏好优化（DPO），面临分布偏移的挑战。现有方法通常通过从目标策略进行在线采样来解决该问题，这需要大量计算资源。本文提出假设：在离策略训练过程中，尽管策略生成输出的排序顺序发生变化，但其整体分布保持相对稳定。这种稳定性使得将目标策略的采样过程转化为计算高效的偏好数据重排序成为可能。基于该假设，我们提出一个新框架，利用模型内在的安全判断能力提取奖励信号，进而计算标签置信度以用于偏好重排序。大量实验与理论分析表明，所提方法有效解决了分布偏移问题，显著提升了安全性能，同时避免了约300倍的计算开销。
英文摘要：
Reinforcement Learning (RL) algorithms for safety alignment of Large Language Models (LLMs), such as Direct Preference Optimization (DPO), encounter the challenge of distribution shift. Current approaches typically address this issue through online sampling from the target policy, which requires significant computational resources. In this paper, we hypothesize that during off-policy training, while the ranking order of output generated by policy changes, their overall distribution remains relatively stable. This stability allows the conversion of the sampling process from the target policy into a computationally efficient re-ranking of preference data. Building on this hypothesis, we propose a new framework that leverages the model's intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. Extensive experiments and theoretical analysis demonstrate that the proposed method effectively addresses the distribution shift issue, remarkably enhancing the safety performance while avoiding about 300x computational overheads.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
