arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.02790cs.CLcs.AI
收藏

分割文本并学习其奖励以改进语言模型中的 RLHF

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Microsoft(微软公司)

Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, Mingyuan Zhou

AI总结:

本文提出片段级奖励模型,通过动态文本分割、位置感知奖励归一化与插值稠密化,缓解 RLHF 中整句奖励稀疏和 token 级奖励过细的问题。

摘要:

基于人类反馈的强化学习(RLHF)已被广泛用于使语言模型(LM)与人类偏好对齐。以往的 RLHF 工作通常采用老虎机(bandit)形式,这种方式虽然直观,却忽视了 LM 生成的序列性质,并可能受到稀疏奖励问题的影响。尽管近期工作提出了稠密的 token 级 RLHF,但将每个 token 视为一个动作可能过于细碎,难以进行恰当的奖励分配。本文中,我们试图通过训练并利用一个片段级奖励模型来兼得两者的优势,该模型为每个语义完整、跨越短 token 序列的文本片段分配奖励。在奖励学习方面,我们的方法支持动态文本分割,并兼容标准的序列偏好数据集。为了依据片段奖励进行有效的基于 RL 的 LM 训练,我们将经典的标量 bandit 奖励归一化器推广为位置感知的归一化器函数,并对片段奖励进行插值以进一步稠密化。借助这些设计,我们的方法在三个流行的 LM 策略 RLHF 基准上表现出竞争力:AlpacaEval 2.0、Arena-Hard 和 MT-Bench。我们还开展了消融研究以进一步证明该方法的有效性。

英文摘要:

Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Prior RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. With these designs, our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑