arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.20722cs.AI
收藏

缓冲区很重要:在大语言模型推理中释放离策略强化学习的潜力

Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun

AI总结:

本文提出BAPO框架,通过动态选择训练批次和保证策略改进下限,提升大语言模型训练效率,实验显示在数学、规划和视觉推理任务中平均提升12.5%。

摘要:

传统基于策略的强化学习框架在验证奖励方面面临经验浪费和奖励同质化问题,这直接阻碍了大语言模型在困难样本上的学习效率。本文介绍Batch Adaptation Policy Optimization (BAPO),一种离策略RLVR框架,旨在提高大语言模型训练效率。它通过重新评估历史上困难的样本并重用高质量样本动态选择训练批次,同时保持策略改进的下限保证。广泛实验进一步证明,BAPO在数学、规划和视觉推理任务中平均比GRPO提升12.5%。关键的是,BAPO成功解决了基础模型始终无法解决的40.7%的问题。

英文摘要:

Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑