arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.06452cs.LGcs.AIcs.CL
收藏

理解RLHF对LLM泛化能力与多样性的影响

Understanding the Effects of RLHF on LLM Generalisation and Diversity

University College London(伦敦大学学院);Meta;University of Oxford(牛津大学)

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu

AI总结:

本文分析了RLHF各阶段对LLM分布外泛化能力和输出多样性的影响,发现RLHF比SFT泛化更好但会显著降低输出多样性,揭示了两者间的权衡关系。

摘要:

使用人类反馈强化学习(RLHF)微调的大型语言模型(LLMs)已被用于目前部署最广泛的一些AI模型中,例如OpenAI的ChatGPT或Anthropic的Claude。尽管在开发这些方法方面已有大量工作,但我们对RLHF各个阶段的优缺点仍然了解有限。为了填补这一空白,我们对流程中每个阶段(即监督微调(SFT)、奖励建模和RLHF)如何影响两个关键属性进行了广泛分析:分布外(OOD)泛化和输出多样性。考虑到这些模型被用于广泛的现实场景,OOD泛化至关重要;而输出多样性指模型生成多样化输出的能力,对多种用例很重要。我们在两个基础模型上,针对摘要和指令遵循任务进行了分析,后者与当前LLM用例高度相关。我们发现,RLHF比SFT更能泛化到新输入,特别是当训练集与测试集之间的分布偏移变大时。然而,RLHF在多种度量下显著降低了输出多样性,这意味着当前LLM微调方法在泛化与多样性之间存在权衡。我们的结果为根据应用选择微调方法提供了指导,并表明需要更多研究来改善泛化与多样性之间的权衡。

英文摘要:

Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date, such as OpenAI's ChatGPT or Anthropic's Claude. While there has been significant work developing these methods, our understanding of the benefits and downsides of each stage in RLHF is still limited. To fill this gap, we present an extensive analysis of how each stage of the process (i.e. supervised fine-tuning (SFT), reward modelling, and RLHF) affects two key properties: out-of-distribution (OOD) generalisation and output diversity. OOD generalisation is crucial given the wide range of real-world scenarios in which these models are being used, while output diversity refers to the model's ability to generate varied outputs and is important for a variety of use cases. We perform our analysis across two base models on both summarisation and instruction following tasks, the latter being highly relevant for current LLM use cases. We find that RLHF generalises better than SFT to new inputs, particularly as the distribution shift between train and test becomes larger. However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity. Our results provide guidance on which fine-tuning method should be used depending on the application, and show that more research is needed to improve the tradeoff between generalisation and diversity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑