迈向可靠且不确定性感知的对齐方法
Towards Reliable, Uncertainty-Aware Alignment
AI总结：
本文针对奖励模型训练不稳定导致对齐性能下降的问题，提出方差感知策略优化框架，通过引入融合奖励模型方差估计的正则化器，可证明地降低输出更差策略的风险，并在多种配置下实现更稳定鲁棒的对齐。
摘要：
大语言模型（LLMs）的对齐通常涉及在偏好数据上训练奖励模型，随后针对该奖励模型进行策略优化。然而，针对单一奖励模型估计进行策略优化可能使其容易受到奖励模型不准确性的影响。我们在开源基准上对奖励模型训练的可变性进行了实证研究。我们观察到，在相同偏好数据集上独立训练的奖励模型可能表现出显著的不一致，这凸显了当前对齐策略的不稳定性。借助一个理论模型，我们证明奖励模型估计中的可变性可能导致过拟合，从而带来性能下降的风险。为缓解这一风险，我们提出了一种用于基于偏好的对齐的方差感知策略优化框架。该框架的关键组成部分是一个新的策略正则化器，它融合了奖励模型的方差估计。我们证明，方差感知策略优化可证明地降低了输出比默认策略更差策略的风险。跨多种大语言模型和奖励模型配置的实验证实，与标准（方差不感知）流程相比，我们的方法能产生更稳定、更鲁棒的对齐效果。
英文摘要：
Alignment of large language models (LLMs) typically involves training a reward model on preference data, followed by policy optimization with respect to the reward model. However, optimizing policies with respect to a single reward model estimate can render it vulnerable to inaccuracies in the reward model. We empirically study the variability of reward model training on open-source benchmarks. We observe that independently trained reward models on the same preference dataset can exhibit substantial disagreement, highlighting the instability of current alignment strategies. Employing a theoretical model, we demonstrate that variability in reward model estimation can cause overfitting, leading to the risk of performance degradation. To mitigate this risk, we propose a variance-aware policy optimization framework for preference-based alignment. The key ingredient of the framework is a new policy regularizer that incorporates reward model variance estimates. We show that variance-aware policy optimization provably reduces the risk of outputting a worse policy than the default. Experiments across diverse LLM and reward model configurations confirm that our approach yields more stable and robust alignment than the standard (variance-unaware) pipeline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
