通过重采样与经验回放实现方差缩减
Variance Reduction via Resampling and Experience Replay
AI总结：
本文针对经验回放理论特性不明的问题，构建基于重采样U/V统计量的理论框架给出方差缩减保证，验证其在多类机器学习任务中的降本增效效果。
摘要：
经验回放是强化学习中的一项基础技术，它通过将过往经验存储在回放缓冲区中并在训练阶段重复使用这些经验，提升学习过程的稳定性。尽管该技术在实践中取得了成功，其理论特性仍未得到充分探究。本文提出一个利用重采样U统计量与V统计量对经验回放进行建模的理论框架，给出了严格的方差缩减保证。我们将该框架应用于采用最小二乘时序差分（LSTD）算法、基于偏微分方程（PDE）的无模型算法的策略评估任务中，证明其可显著提升稳定性与效率，在数据稀缺场景下效果尤为突出。除策略评估外，我们将该框架拓展至核岭回归任务，证明基于经验回放的方法可将传统方法的时间复杂度从O(n³)降至最低O(n²)，同时实现方差缩减。大量数值实验验证了我们的理论结论，证明经验回放在各类机器学习任务中具备广泛适用性与有效性。
英文摘要：
Experience replay is a foundational technique in reinforcement learning that enhances learning stability by storing past experiences in a replay buffer and reusing them during training. Despite its practical success, its theoretical properties remain underexplored. In this paper, we present a theoretical framework that models experience replay using resampled $U$- and $V$-statistics, providing rigorous variance reduction guarantees. We apply this framework to policy evaluation tasks using the Least-Squares Temporal Difference (LSTD) algorithm and a Partial Differential Equation (PDE)-based model-free algorithm, demonstrating significant improvements in stability and efficiency, particularly in data-scarce scenarios. Beyond policy evaluation, we extend the framework to kernel ridge regression, showing that the experience replay-based method reduces the computational cost from the traditional $O(n^3)$ in time to as low as $O(n^2)$ in time while simultaneously reducing variance. Extensive numerical experiments validate our theoretical findings, demonstrating the broad applicability and effectiveness of experience replay in diverse machine learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
