arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节（Veterans Day），12 日无更新
arXiv 2310.09520cs.CL
收藏

奖励增强解码：基于单向奖励模型的高效受控文本生成

Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model

UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）

Haikang Deng, Colin Raffel

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Reward-Augmented Decoding (RAD)，利用小型单向奖励模型在解码时对生成文本评分并调整采样概率，以高效实现受控文本生成，其性能媲美需重新训练的最先进方法且计算开销极小。

摘要：

尽管大型语言模型在大量下游应用中证明了其有效性，但它们经常生成有问题的或缺乏所需属性的文本。在本文中，我们引入了Reward-Augmented Decoding (RAD)（奖励增强解码），这是一种文本生成过程，它使用小型单向奖励模型来鼓励语言模型生成具有特定属性的文本。具体而言，RAD使用奖励模型对生成过程中的文本进行评分，并重新调整采样概率以偏向高奖励的token。通过使用单向奖励模型，RAD能够缓存先前生成步骤的激活，从而降低计算开销。通过在生成无毒文本和情感受控文本上的实验，我们证明RAD在仅改变生成过程的方法中表现最佳，并且匹配了涉及重新训练语言模型的最先进方法的性能。我们进一步验证了RAD在超大型语言模型上有效，同时仅产生最小的计算开销。

英文摘要：

While large language models have proven effective in a huge range of downstream applications, they often generate text that is problematic or lacks a desired attribute. In this paper, we introduce Reward-Augmented Decoding (RAD), a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text that has certain properties. Specifically, RAD uses the reward model to score generations as they are produced and rescales sampling probabilities to favor high-reward tokens. By using a unidirectional reward model, RAD can cache activations from prior generation steps to decrease computational overhead. Through experiments on generating non-toxic and sentiment-controlled text, we demonstrate that RAD performs best among methods that change only the generation procedure and matches the performance of state-of-the-art methods that involve re-training the language model. We further validate that RAD is effective on very large language models while incurring a minimal computational overhead.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑