arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.14718cs.CL
收藏

Leftover Lunch:基于优势的离线强化学习用于语言模型

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

Georgia Institute of Technology(佐治亚理工学院);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学)

Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, Mark Riedl

AI总结:

针对RLHF不稳定且数据消耗大的问题,提出离线策略梯度算法A-LoL,通过序列级奖励和正优势数据训练,实现高效稳定的LM对齐,在多项任务中优于基线。

摘要:

基于人类反馈的强化学习(RLHF)是语言模型(LM)对齐最突出的方法。然而,RLHF是一个不稳定且数据消耗大的过程,不断需要新的高质量LM生成数据进行微调。我们引入了Advantage-Leftover Lunch RL(A-LoL),这是一类新的离线策略梯度算法,能够在任何已有数据上进行RL训练。通过将整个LM输出序列视为单个动作,A-LoL允许将序列级分类器或人工设计的评分函数作为奖励。随后,通过使用LM的价值估计,A-LoL仅在正优势(leftover)数据点上进行训练,使其对噪声具有鲁棒性。总体而言,A-LoL是一种易于实现、样本高效且稳定的LM训练方案。我们通过四项不同的语言生成任务展示了A-LoL及其变体的有效性。我们与在线RL(PPO)以及近期基于偏好的(DPO、PRO)和基于奖励的(GOLD)离线RL基线进行了比较。在常用的RLHF基准Helpful and Harmless Assistant(HHA)上,使用A-LoL方法训练的LM实现了最高的多样性,同时根据人类评估,其安全性和有用性也高于基线。此外,在其余三项任务中,即使使用噪声或次优训练数据,A-LoL仍能优化多个不同的奖励函数。我们还发布了实验代码。https://github.com/abahati95/LoL-RL

英文摘要:

Reinforcement Learning with Human Feedback (RLHF) is the most prominent method for Language Model (LM) alignment. However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning. We introduce Advantage-Leftover Lunch RL (A-LoL), a new class of offline policy gradient algorithms that enable RL training on any pre-existing data. By assuming the entire LM output sequence as a single action, A-LoL allows incorporating sequence-level classifiers or human-designed scoring functions as rewards. Subsequently, by using LM's value estimate, A-LoL only trains on positive advantage (leftover) data points, making it resilient to noise. Overall, A-LoL is an easy-to-implement, sample-efficient, and stable LM training recipe. We demonstrate the effectiveness of A-LoL and its variants with a set of four different language generation tasks. We compare against both online RL (PPO) and recent preference-based (DPO, PRO) and reward-based (GOLD) offline RL baselines. On the commonly-used RLHF benchmark, Helpful and Harmless Assistant (HHA), LMs trained with A-LoL methods achieve the highest diversity while also being rated more safe and helpful than the baselines according to humans. Additionally, in the remaining three tasks, A-LoL could optimize multiple distinct reward functions even when using noisy or suboptimal training data. We also release our experimental code. https://github.com/abaheti95/LoL-RL

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑