为大语言模型高效训练的体验回放
Efficient RL Training for LLMs with Experience Replay
AI总结：
本文研究了大语言模型训练中体验回放的优化设计，通过平衡陈旧性方差、样本多样性与生成成本，证明了严格策略采样在生成成本高时效果不佳，实验表明合理设计的回放缓冲器能显著降低推理计算成本而不影响模型性能。
摘要：
尽管体验回放——在训练过程中存储和多次重用轨迹——是通用强化学习中的基础技术，但在大语言模型训练后仍鲜有研究，由于普遍认为新鲜的策略数据对高性能至关重要。本文挑战这一假设。我们系统研究了大语言模型训练后的回放缓冲器，将最优设计形式化为陈旧性导致的方差、样本多样性和生成的高计算成本之间的权衡。我们证明当生成成本高时，严格策略采样是次优的。实证上，我们证明了精心设计的回放缓冲器可以显著降低推理计算成本，而不降低甚至在某些情况下提高最终模型性能，同时保持策略熵。
英文摘要：
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
