Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学)
NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
AIML, University of Adelaide(阿德莱德大学AIML);Adobe Research(奥多比研究院);University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Santa Cruz(加利福尼亚大学圣克鲁兹分校)
Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院);Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance(北京多模态数据智能感知与治理重点实验室)
Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
Tsinghua University(清华大学);University of Hamburg(汉堡大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Qi Zhi Institute(上海期智研究院);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院)
Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
Army Research Laboratory(陆军研究实验室);Washington State University(华盛顿州立大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Sony AI(索尼AI);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究所)
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Department of Electrical and Computer Engineering, Seoul National University(首尔国立大学电子与计算机工程系);Interdisciplinary Program in AI, Seoul National University(首尔国立大学人工智能跨学科项目);AIIS, ASRI, INMC, and ISRC, Seoul National University(首尔国立大学人工智能研究所、人工智能研究机构、智能网络与计算中心及人工智能科学研究中心)
Imam Mohammad Ibn Saud Islamic University (IMSIU)(伊玛姆·穆罕默德·本·沙特伊斯兰大学);Monash University(莫纳什大学);University of New South Wales, Sydney(新南威尔士大学,悉尼);Cerenaut
Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
Facebook AI Research(脸书人工智能研究院);McGill University(麦吉尔大学)
非策略深度强化学习(RL)算法无法在不与环境进行在线交互的情况下仅从批量离线数据中学习,这是由于被称为\textit{extrapolation error}的现象。这通常是由于重放缓冲区中可用的过去数据可能与当前策略下的数据分布完全不同。我们认为,大多数离策略学习方法从根本上遭受\textit{state distribution shift}由于行为和目标政策收集的数据的状态访问分布之间的不匹配。当前样本和过去样本之间的这种数据分布偏移会显著影响大多数现代的基于脱离策略的策略优化算法的性能。本文首先系统地分析了离策略学习中的状态分布失配问题,然后提出了一种约束状态分布偏移的离线策略优化方法。为此,我们首先使用密度估计器基于状态的特征来估计状态分布,然后开发一种新的约束非策略梯度目标,其最小化状态分布偏移。我们的连续控制任务的实验结果表明,最大限度地减少这种分布不匹配,可以显着提高性能,在最流行的实用的政策外策略梯度算法。