通过二次回滚实现更好的RL训练数据利用
Towards Better RL Training Data Utilization via Second-Order Rollout
AI总结：
本文提出通过二次回滚提升RL训练数据利用效率，联合训练生成与批判能力，实验表明在相同数据下性能更优，并揭示标签平衡与噪声问题的解决方法。
摘要：
强化学习（RL）已使大型语言模型（LLMs）具备强大的推理能力，但传统的RL主要通过仅使用一阶回滚（为一个问题生成多个响应）来提升生成能力，我们认为这种方法未能充分利用训练数据的潜力，因为忽略了批判能力的训练。为解决这个问题，我们进一步引入了二次回滚（为一个响应生成多个批判）的概念，并提出一个联合训练生成和批判能力的统一框架。在各种模型和数据集上的广泛实验表明，我们的方法比传统的RL更有效地利用训练数据，并在相同训练数据下取得更好的性能。此外，我们揭示了关于二次回滚和批判训练的一些有洞察力的发现，例如批判训练中标签平衡的重要性以及基于结果的奖励的噪声问题，这些问题可以通过采样技术来缓解。我们的工作对RL中的动态数据增强和联合生成-批判训练进行了初步探索，为RL训练的进一步发展提供了有意义的启发。
英文摘要：
Reinforcement Learning (RL) has empowered Large Language Models (LLMs) with strong reasoning capabilities, but vanilla RL mainly focuses on generation capability improvement by training with only first-order rollout (generating multiple responses for a question), and we argue that this approach fails to fully exploit the potential of training data because of the neglect of critique capability training. To tackle this problem, we further introduce the concept of second-order rollout (generating multiple critiques for a response) and propose a unified framework for jointly training generation and critique capabilities. Extensive experiments across various models and datasets demonstrate that our approach can utilize training data more effectively than vanilla RL and achieve better performance under the same training data. Additionally, we uncover several insightful findings regarding second-order rollout and critique training, such as the importance of label balance in critique training and the noise problem of outcome-based rewards, which can be mitigated through sampling techniques. Our work offers a preliminary exploration of dynamic data augmentation and joint generation-critique training in RL, providing meaningful inspiration for the further advancement of RL training
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
