崩溃前的繁荣:基于大语言模型陈旧数据的离策略RL能走多远?
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
AI总结:
针对大语言模型同策略强化学习效率低、异步离策略训练因数据陈旧易性能下降或崩溃的问题,提出二阶矩信任策略优化M2PO,通过约束重要性权重二阶矩抑制极端离群值,在高陈旧度下仍能稳定训练且性能匹配同策略方法。
摘要:
强化学习一直是大语言模型推理能力近期进展的核心,但大多数算法依赖同策略训练,每次更新都需要全新的采样轨迹,这限制了训练效率与可扩展性。异步强化学习系统通过将采样轨迹生成与训练解耦来缓解这一问题,但其有效性取决于能否容忍采样数据的高度陈旧性——在该场景下,现有方法要么性能下降,要么训练崩溃。我们重新审视了这一挑战,并发现了“崩溃前繁荣”现象:若利用得当,陈旧数据可以和同策略数据具备同等信息量。基于这一洞见,我们提出M2PO(二阶矩信任策略优化),该方法通过约束重要性权重的二阶矩,仅抑制极端离群值,同时保留有信息量的更新。值得注意的是,在高陈旧度下,M2PO大幅降低了被裁剪token的比例(训练期间从1.22%降至0.06%),精准屏蔽高方差token的同时维持了优化的稳定性。我们在6个模型(参数规模从1.7B到32B)和8个基准上开展了广泛评估,结果表明,即使数据至少陈旧了256次模型更新,M2PO仍能实现稳定的离策略训练,且性能与同策略训练相当。
英文摘要:
Reinforcement learning has been central to recent advances in large language model reasoning, but most algorithms rely on on-policy training that demands fresh rollouts at every update, limiting efficiency and scalability. Asynchronous RL systems alleviate this by decoupling rollout generation from training, yet their effectiveness hinges on tolerating large staleness in rollout data, a setting where existing methods either degrade in performance or collapse. We revisit this challenge and uncover a prosperity-before-collapse phenomenon: stale data can be as informative as on-policy data if exploited properly. Building on this insight, we introduce M2PO (Second-Moment Trust Policy Optimization), which constrains the second moment of importance weights to suppress only extreme outliers while preserving informative updates. Notably, M2PO sharply reduces the fraction of clipped tokens under high staleness (from 1.22% to 0.06% over training), precisely masking high-variance tokens while maintaining stable optimization. Extensive evaluation across six models (from 1.7B to 32B) and eight benchmarks shows that M2PO delivers stable off-policy training even with data stale by at least 256 model updates and matches on-policy performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。