异步代理强化学习中缺失旧日志：语义不匹配及用于离线策略修正的修复方法
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
AI总结：
本文研究了异步代理强化学习中因缺失旧日志导致的语义不匹配问题，提出三种精确获取旧日志的策略及近似修正方法，改进了PPO-EWMA方法，提升了训练速度和优化性能。
摘要：
异步强化学习通过将样本生成与策略优化解耦，提高了大语言模型代理的回放吞吐量，但同时也引入了PPO类离线策略修正中的关键故障模式。在异构训练系统中，总重要性比率应理想地分解为两个语义不同的因素：一个训练-推理不匹配项，用于对齐同一行为策略版本的推理侧和训练侧分布，以及一个策略陈旧项，用于约束从历史策略到当前策略的更新。我们发现实际的异步管道在延迟更新和部分回放的情况下，常常丢失所需的训练侧旧日志或旧日志。这种缺失旧日志的问题使不匹配修复与陈旧修正纠缠在一起，破坏了解耦修正的初衷，并使裁剪和掩码阈值产生不良交互。为了解决这一问题，我们研究了精确和近似修正路径。我们提出了三种精确旧日志获取策略：基于快照的版本跟踪、专用旧日志模型以及通过部分回放中断进行同步，并比较了它们的系统权衡。从近似修正的角度来看，我们关注通过更合适的近似策略保留解耦修正的好处，当无法以低成本恢复精确旧日志时，不增加额外系统开销。随后，我们采用改进的PPO-EWMA方法，该方法在训练速度和优化性能方面均取得显著提升。
英文摘要：
Asynchronous reinforcement learning improves rollout throughput for large language model agents by decoupling sample generation from policy optimization, but it also introduces a critical failure mode for PPO-style off-policy correction. In heterogeneous training systems, the total importance ratio should ideally be decomposed into two semantically distinct factors: a \emph{training--inference discrepancy term} that aligns inference-side and training-side distributions at the same behavior-policy version, and a \emph{policy-staleness term} that constrains the update from the historical policy to the current policy. We show that practical asynchronous pipelines with delayed updates and partial rollouts often lose the required historical training-side logits, or old logits. This missing-old-logit problem entangles discrepancy repair with staleness correction, breaks the intended semantics of decoupled correction, and makes clipping and masking thresholds interact undesirably. To address this issue, we study both exact and approximate correction routes. We propose three exact old-logit acquisition strategies: snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption, and compare their system trade-offs. From the perspective of approximate correction, we focus on preserving the benefits of decoupled correction through a more appropriate approximate policy when exact old logits cannot be recovered at low cost, without incurring extra system overhead. Following this analysis, we adopt a revised PPO-EWMA method, which achieves significant gains in both training speed and optimization performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
