基于奖励偏移和截断KL正则化的RLHF泛化
Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
AI总结：
本文提出基于奖励偏移和截断KL正则化的RLHF泛化理论，分析了奖励偏移和KL截断对泛化误差的影响，并给出了优化KL截断阈值和预算分配的实践指导。
摘要：
大型语言模型中的对齐和适应严重依赖于人类反馈强化学习（RLHF）；然而，其泛化能力的理论理解仍不充分，尤其是在学习的奖励可能偏移，且KL控制被估计和截断的情况下。为解决这一问题，我们开发了RLHF的泛化理论，明确考虑了（1）奖励偏移：奖励模型在早期或混合行为策略上的偏好数据上进行训练，而RLHF在自身回放上优化当前策略；以及（2）截断KL正则化：KL正则化器从采样对数概率比中估计，然后截断以稳定，导致RLHF的误差。我们为RLHF提出了泛化界限，表明泛化误差源于提示和回放的采样误差、奖励偏移误差和KL截断误差。我们还讨论了特殊情况（1）使用有限空间上的均匀先验初始化RLHF参数，以及（2）通过随机梯度下降训练RLHF，作为奥尔森-乌尔岑贝克过程。该理论在（1）最优KL截断阈值和（2）提示、回放和偏好数据预算分配方面具有实际意义。
英文摘要：
Alignment and adaptation in large language models heavily rely on reinforcement learning from human feedback (RLHF); yet, theoretical understanding of its generalisability remains premature, especially when the learned reward could shift, and the KL control is estimated and clipped. To address this issue, we develop generalisation theory for RLHF that explicitly accounts for (1) \emph{reward shift}: reward models are trained on preference data from earlier or mixed behaviour policies while RLHF optimises the current policy on its own rollouts; and (2) \emph{clipped KL regularisation}: the KL regulariser is estimated from sampled log-probability ratios and then clipped for stabilisation, resulting in an error to RLHF. We present generalisation bounds for RLHF, suggesting that the generalisation error stems from a sampling error from prompts and rollouts, a reward shift error, and a KL clipping error. We also discuss special cases of (1) initialising RLHF parameters with a uniform prior over a finite space, and (2) training RLHF by stochastic gradient descent, as an Ornstein-Uhlenbeck process. The theory yields practical implications in (1) optimal KL clipping threshold, and (2) budget allocation in prompts, rollouts, and preference data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
