arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.08714cs.CLcs.AIcs.LG
收藏

条件文本生成中的奖励博弈

Reward Gaming in Conditional Text Generation

New York University(纽约大学);Google DeepMind(谷歌DeepMind)

Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, He He

AI总结:

本文揭示条件文本生成RL训练中三类奖励误配导致的奖励博弈问题,分析其放大机制并提出修复方向。

摘要:

为了使条件文本生成模型的输出与期望行为保持一致,人们越来越关注利用从人类标注中学习到的奖励函数,通过强化学习(RL)来训练模型。在这一框架下,我们识别出三种高奖励被错误分配给不良模式的常见情形:噪声诱发的虚假相关、自然产生的虚假相关以及协变量偏移。我们表明,尽管学习到的指标在用于训练奖励函数的数据分布上取得了高性能,但在文本生成模型的RL训练过程中,这些不良模式可能会被放大。虽然RL或安全领域已经讨论过奖励博弈,但在这篇讨论文中,我们希望通过具体的条件文本生成示例,提醒自然语言生成(NLG)领域关注奖励博弈问题,并讨论可能的修复方法及未来工作方向。

英文摘要:

To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑