arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07780cs.LGcs.CL
收藏

对Token级奖励引导文本生成的批判性审视

A Critical Look At Tokenwise Reward-Guided Text Generation

University of Waterloo(滑铁卢大学);Vector Institute(向量研究所);University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心);Western University(西方大学)

Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, Pascal Poupart

AI总结:

针对Token级奖励引导文本生成,本文指出全序列奖励模型不适用于部分序列评分,提出在部分序列上训练Bradley-Terry模型并自回归采样,其策略与两个RLHF策略比率成正比,性能优于先前方法且无需大规模微调。

摘要:

大型语言模型(LLMs)可以通过微调与人类偏好对齐来改进——即所谓的基于人类反馈的强化学习(RLHF)。然而,对许多用户而言,微调LLM的成本高得令人望而却步。由于能够绕过LLM微调,预测时Token级奖励引导文本生成(RGTG)方法最近被提出。这些方法使用在完整序列上训练的奖励模型,在解码过程中对部分序列进行评分,旨在引导生成朝向高奖励的序列。然而,这些方法迄今为止仅基于启发式动机,且分析不足。在这项工作中,我们表明在完整序列上训练的奖励模型与对部分序列评分不兼容。为缓解这一问题,我们提出显式地在部分序列上训练Bradley-Terry奖励模型,并在解码过程中自回归地从隐含的Token级策略中采样。我们研究了该奖励模型及所得策略的性质:我们证明该策略与两个不同RLHF策略的比率成正比。我们的简单方法优于先前的RGTG方法,并且在没有大规模LLM微调的情况下,性能与强离线基线相当。我们工作的代码可在https://github.com/ahmadrash/PARGS获取。

英文摘要:

Large language models (LLMs) can be improved by aligning with human preferences through fine-tuning -- the so-called reinforcement learning from human feedback (RLHF). However, the cost of fine-tuning an LLM is prohibitive for many users. Due to their ability to bypass LLM fine-tuning, prediction-time tokenwise reward-guided text generation (RGTG) methods have recently been proposed. They use a reward model trained on full sequences to score partial sequences during decoding in a bid to steer the generation towards sequences with high rewards. However, these methods have so far been only heuristically motivated and poorly analyzed. In this work, we show that reward models trained on full sequences are not compatible with scoring partial sequences. To alleviate this, we propose to train a Bradley-Terry reward model on partial sequences explicitly, and autoregressively sample from the implied tokenwise policy during decoding. We study the properties of this reward model and the resulting policy: we show that this policy is proportional to the ratio of two distinct RLHF policies. Our simple approach outperforms previous RGTG methods and performs similarly to strong offline baselines without large-scale LLM fine-tuning. Code for our work is available at https://github.com/ahmadrash/PARGS

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑