arXiv 2604.169182026-04-18cs.CLcs.LG

具有新鲜度意识的优先经验回放用于LLM/VLM强化学习

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning

King Abdullah University of Science and Technology（卡布尔大学科学与技术大学）；Chinese Academy of Sciences, Institute of Automation（中国科学院自动化研究所）；AI Centre, Department of Computer Science, University College London（伦敦大学学院人工智能中心）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究所）

Weiyu Ma, Yongcheng Zeng, Yan Song, Xinyu Cui, Jian Zhao, Xuhui Liu, Mohamed Elhoseiny

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Freshness-Aware PER，通过引入指数衰减机制解决LLM/VLM强化学习中优先级过时问题，显著提升样本效率，在多个任务中取得优异表现。

摘要：

强化学习（RL）在训练大型语言模型（LLMs）和视觉-语言模型（VLMs）后取得了显著成功，单策略算法如PPO、GRPO和REINFORCE++成为主流范式。然而，这些方法在单次梯度更新后丢弃所有收集的轨迹，导致样本效率低下，尤其在代理任务中，多轮环境交互成本高昂。经验回放通过允许智能体重用过去轨迹并优先选择信息量大的轨迹来提升样本效率，但直接将优先经验回放（PER）应用于LLMs失效。百亿参数模型的快速策略演变使存储的优先级过时，导致旧的高优先级轨迹在变得无信息后仍主导采样。本文提出Freshness-Aware PER，通过在任何基于PER的优先级上加入基于有效样本量分析的乘法指数年龄衰减机制，解决优先级过时问题。本文是首个成功将PER应用于LLM/VLM强化学习的工作。我们在八个多步骤代理、推理和数学竞赛任务上进行了评估，使用0.5B、3B和7B模型。Freshness-Aware PER显著优于单策略基线，在NQ Search上提升46%，在Sokoban上提升367%，在VLM FrozenLake上提升133%，而标准PER无年龄衰减则持续恶化性能。我们的代码在https://github.com/Vision-CAIR/Freshness-Aware-PER公开。

英文摘要：

Reinforcement Learning (RL) has achieved impressive success in post-training Large Language Models (LLMs) and Vision-Language Models (VLMs), with on-policy algorithms such as PPO, GRPO, and REINFORCE++ serving as the dominant paradigm. However, these methods discard all collected trajectories after a single gradient update, resulting in poor sample efficiency, particularly wasteful for agentic tasks where multi-turn environment interactions are expensive. While Experience Replay drives sample efficiency in classic RL by allowing agents to reuse past trajectories and prioritize informative ones, directly applying Prioritized Experience Replay (PER) to LLMs fails. The rapid policy evolution of billion-parameter models renders stored priorities stale, causing old high-priority trajectories to dominate sampling long after they have become uninformative. We propose Freshness-Aware PER, which addresses this priority staleness problem by augmenting any PER-based priority with a multiplicative exponential age decay grounded in effective sample size analysis. To the best of our knowledge, Freshness-Aware PER is the first work to successfully apply PER to LLM/VLM reinforcement learning. We evaluate on eight multi-step agentic, reasoning, and math competition tasks with 0.5B, 3B, and 7B models. Freshness-Aware PER significantly outperforms on-policy baselines, achieving +46% on NQ Search, +367% on Sokoban, and +133% on VLM FrozenLake, while standard PER without age decay consistently degrades performance. Our code is publicly available at https://github.com/Vision-CAIR/Freshness-Aware-PER.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。