arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.17031cs.LG
收藏

RLHF结合PPO的N+实现细节:以TL;DR摘要生成为案例研究

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Hugging Face;Mila(米拉实验室);Université de Montréal(蒙特利尔大学);Fuxi AI Lab(伏羲AI实验室);NetEase(网易)

Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, Lewis Tunstall

AI总结:

本研究首次公开复现OpenAI TL;DR摘要工作的RLHF缩放行为,从零构建流水线并总结20余个关键实现细节,其训练的Pythia模型在响应质量上随规模扩展显著提升,2.8B和6.9B模型超越OpenAI的1.3B checkpoint,并开源代码与模型。

摘要:

本研究首次公开复现了OpenAI开创性TL;DR摘要工作中报告的Reinforcement Learning from Human Feedback (RLHF)缩放行为。我们从零构建RLHF流水线,列举超过20个关键实现细节,并分享复现过程中的核心见解。经RLHF训练的Pythia模型在响应质量上展现出随模型规模扩展的显著提升,其中2.8B和6.9B模型优于OpenAI发布的1.3B checkpoint。我们公开发布训练好的模型checkpoints和代码,以促进后续研究并加速该领域进展。

英文摘要:

This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field (\url{https://github.com/vwxyzjn/summarize_from_feedback_details}).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑