在策略优化中经验回放的收敛性：刻画偏差、方差和有限时间收敛
On the Convergence of Experience Replay in Policy Optimization: Characterizing Bias, Variance, and Finite-Time Convergence
AI总结：
本文提出了一种理论框架，分析经验回放中偏差、方差和有限时间收敛性，揭示缓冲大小与样本相关性对策略优化收敛率的影响。
摘要：
经验回放是现代深度强化学习的核心组成部分，然而其在策略优化中的益处在经验启发法之外的理解甚微。本文为现代策略梯度方法的经验回放开发了一种新的理论框架，其中两种来源的依赖性从根本上复杂化了分析：轨迹上的马尔可夫相关性和优化迭代间的策略漂移。我们引入了一种基于辅助马尔可夫链和滞后解耦的新证明技术，使这些依赖性变得可处理。在该框架下，我们推导出在回放下的策略梯度估计器的有限时间偏差界，识别偏差如何随累积策略更新、底层动态的混合时间和缓冲数据的年龄而变化，从而正式化实践者避免过于陈旧回放的规则。我们进一步提供了一种相关性感知的方差分解，展示样本依赖性如何支配回放的梯度方差以及回放何时有益。基于这些特征化，我们建立了基于经验回放的策略优化的有限时间收敛保证，明确量化缓冲大小、样本相关性和混合如何共同决定收敛率，并揭示了一个固有的偏差-方差权衡：更大的缓冲区可以减少方差通过平均不相关的样本，但会增加偏差因为数据变得陈旧。这些结果为缓冲区大小和回放计划提供了原则性的指导，将先前的经验发现与定量理论联系起来。
英文摘要：
Experience replay is a core ingredient of modern deep reinforcement learning, yet its benefits in policy optimization are poorly understood beyond empirical heuristics. This paper develops a novel theoretical framework for experience replay in modern policy gradient methods, where two sources of dependence fundamentally complicate analysis: Markovian correlations along trajectories and policy drift across optimization iterations. We introduce a new proof technique based on auxiliary Markov chains and lag-based decoupling that makes these dependencies tractable. Within this framework, we derive finite-time bias bounds for policy-gradient estimators under replay, identifying how bias scales with the cumulative policy update, the mixing time of the underlying dynamics, and the age of buffered data, thereby formalizing the practitioner's rule of avoiding overly stale replay. We further provide a correlation-aware variance decomposition showing how sample dependence governs gradient variance from replay and when replay is beneficial. Building on these characterizations, we establish the finite-time convergence guarantees for experience-replay-based policy optimization, explicitly quantifying how buffer size, sample correlation, and mixing jointly determine the convergence rate and revealing an inherent bias-variance trade-off: larger buffers can reduce variance by averaging less correlated samples but can increase bias as data become stale. These results offer a principled guide for buffer sizing and replay schedules, bridging prior empirical findings with quantitative theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
