基于方差减少的经验回放的策略优化
Variance Reduction Based Experience Replay for Policy Optimization
AI总结：
本文提出VRER框架，通过选择性重用信息样本减少策略梯度估计的方差，从而提升策略优化效率。
摘要：
有效的强化学习（RL）对于复杂随机系统需要利用以前迭代中收集的历史数据来加速策略优化。经典的经验回放将所有过去的观察视为均匀，并未能考虑它们对学习的不同贡献。为克服这一限制，我们提出了方差减少经验回放（VRER），一种有原则的框架，该框架选择性地重用有信息量的样本以减少策略梯度估计的方差。VRER是算法无关的，并且能够无缝整合到现有的策略优化方法中，成为我们样本高效反向策略算法Policy Gradient with VRER（PG-VRER）的基础。受经验回放缺乏严格理论分析的启发，我们开发了一个新的框架，明确捕捉由马尔可夫动态和行为策略交互引入的依赖关系。利用这个框架，我们为PG-VRER建立了有限时间收敛保证，并揭示了一个基本的偏差-方差权衡：重用较早的经验会增加偏差，但同时减少梯度方差。广泛的实证实验表明，VRER一致地加速了策略学习，并在最先进的策略优化算法上提高了性能。
英文摘要：
Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization algorithms
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
