arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.08708cs.LGcs.AIcs.CL
收藏

教师强迫恢复奖励功能的文本生成

Teacher Forcing Recovers Reward Functions for Text Generation

University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所(Amii))

Yongchang Hao, Yuxin Liu, Lili Mou

摘要:

强化学习(RL)已被广泛用于文本生成,以减轻曝光偏差问题或利用非并行数据集。奖励函数在RL训练成功中起着重要作用。然而,以前的奖励函数通常是特定于任务的和稀疏的,限制了RL的使用。在我们的工作中,我们提出了一种任务不可知的方法,直接从教师强迫训练的模型中导出逐步奖励函数。此外,我们还提出了一个简单的修改,用我们的诱导奖励函数来稳定非并行数据集上的RL训练。实验结果表明,我们的方法优于自我训练和奖励回归方法在几个文本生成任务,证实了我们的奖励功能的有效性。

英文摘要:

Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets. The reward function plays an important role in making RL training successful. However, previous reward functions are typically task-specific and sparse, restricting the use of RL. In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing. We additionally propose a simple modification to stabilize the RL training on non-parallel datasets with our induced reward function. Empirical results show that our method outperforms self-training and reward regression methods on several text generation tasks, confirming the effectiveness of our reward function.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑