arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.16403cs.LG
收藏

迈向RLHF泛化性的理论理解

Towards a Theoretical Understanding to the Generalization of RLHF

Beijing Institute of Technology(北京理工大学);Zhongguancun Academy(中关村学院);Renmin University of China(中国人民大学)

Zhaochun Li, Mingyang Yi, Yue Wang, Shisheng Cui, Yong Liu

AI总结:

本文通过算法稳定性框架,在线性奖励模型下构建了RLHF下LLM的泛化理论,证明在特征覆盖条件下策略模型的泛化界为O(n^{-1/2}),并推广到梯度方法参数。

摘要:

强化学习从人类反馈(RLHF)及其变种已成为对大型语言模型对齐人类意图的主要方法。尽管在实践中有效,这些方法在高维设置中的理论泛化性质仍待探索。为此,我们基于算法稳定性框架,在线性奖励模型下构建了RLHF下LLM的泛化理论。与现有基于奖励模型最大似然估计一致性的研究不同,我们的分析是在端到端学习框架下进行的,这与实践一致。具体而言,我们证明在关键的特征覆盖条件下,策略模型的经验最优解具有O(n^{-1/2})量级的泛化界。此外,结果可以推广到通过梯度方法获得的参数,即梯度上升(GA)和随机梯度上升(SGA)。因此,我们认为我们的结果为RLHF后LLM经验观察到的泛化性提供了新的理论证据。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) and its variants have emerged as the dominant approaches for aligning Large Language Models with human intent. While empirically effective, the theoretical generalization properties of these methods in high-dimensional settings remain to be explored. To this end, we build the generalization theory on RLHF of LLMs under the linear reward model, through the framework of algorithmic stability. In contrast to the existing works built upon the consistency of maximum likelihood estimations on reward model, our analysis is presented under an end-to-end learning framework, which is consistent with practice. Concretely, we prove that under a key \textbf{feature coverage} condition, the empirical optima of policy model have a generalization bound of order $\mathcal{O}(n^{-\frac{1}{2}})$. Moreover, the results can be extrapolated to parameters obtained by gradient-based learning algorithms, i.e., Gradient Ascent (GA) and Stochastic Gradient Ascent (SGA). Thus, we argue that our results provide new theoretical evidence for the empirically observed generalization of LLMs after RLHF.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑