arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.01693cs.CL
收藏

细粒度人类反馈为语言模型训练提供更优奖励

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣迭戈分校);MosaicML

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi

AI总结:

本文提出Fine-Grained RLHF框架,通过句子级密集奖励和多类型细粒度人类反馈训练语言模型,在去毒化和长文本问答上提升性能并支持行为定制。

摘要:

语言模型(LMs)经常表现出不良的文本生成行为,包括生成虚假、有害或不相关的输出。基于人类反馈的强化学习(RLHF)——将人类对语言模型输出的偏好判断转化为学习信号——最近在解决这些问题方面展现出前景。然而,这种整体性反馈在长文本输出上传递的信息有限;它没有表明输出的哪些方面影响了用户偏好;例如,哪些部分包含什么类型的错误。在本文中,我们使用细粒度人类反馈(例如,哪个句子是虚假的,哪个子句是不相关的)作为显式训练信号。我们提出了Fine-Grained RLHF,这是一个能够在两个方面实现细粒度奖励函数训练和学习的框架:(1)密度,在每个片段(例如,一个句子)生成后提供奖励;(2)纳入与不同反馈类型(例如,事实不正确、不相关和信息不完整)相关联的多个奖励模型。我们在去毒化和长文本问答任务上进行了实验,以说明使用此类奖励函数进行学习如何带来性能提升,并得到了自动评估和人类评估的支持。此外,我们展示了语言模型行为可以通过细粒度奖励模型的不同组合进行定制。我们在https://FineGrainedRLHF.github.io发布了所有数据、收集的人类反馈和代码。

英文摘要:

Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑