带延迟反馈的后验采样方法用于线性函数近似的强化学习
Posterior Sampling with Delayed Feedback for Reinforcement Learning with Linear Function Approximation
AI总结：
针对强化学习中观测延迟导致性能下降的问题，提出带延迟反馈的后验采样算法Delayed-PSVI及基于朗之万动力学的高效版本Delayed-LPSVI，证明其最优遗憾值边界并验证有效性。
摘要：
近期的强化学习（RL）研究通过利用函数近似缓解样本复杂度障碍以提升性能，已取得显著进展。尽管取得了这些成功，现有可证明高效的算法通常依赖于采取动作后可立即获得反馈的前提。由于未能考虑观测延迟的影响，现实系统的性能可能因遗憾值激增而大幅下降。在本研究中，我们采用后验采样方法来解决带线性函数近似的强化学习中的延迟反馈挑战，后验采样已被证实在多种场景下经验性能优于流行的UCB算法。我们首先提出Delayed-PSVI，这是一种基于乐观值的算法，通过后验采样的噪声扰动有效探索值函数空间。我们首次对带延迟反馈的强化学习后验采样算法进行了分析，并证明在存在未知随机延迟的情况下，我们的算法可达到$\tilde{O}(\tau{d^3H^3 T} + d^2H^2 E[\tau])$的最坏情况遗憾值，其中$E[\tau]$为期望延迟。为进一步提升其计算效率并拓展其在高维强化学习问题中的适用性，我们为Delayed-LPSVI引入了一种基于朗之万动力学的梯度近似采样方案，该方案在保持同阶最优遗憾值保证的同时，仅需$\tilde{O}(dHK)$的计算成本。我们开展了实证评估，验证了所提算法在统计性能和计算效率上的有效性。
英文摘要：
Recent studies in reinforcement learning (RL) have made significant progress by leveraging function approximation to alleviate the sample complexity hurdle for better performance. Despite the success, existing provably efficient algorithms typically rely on the accessibility of immediate feedback upon taking actions. The failure to account for the impact of delay in observations can significantly degrade the performance of real-world systems due to the regret blow-up. In this work, we tackle the challenge of delayed feedback in RL with linear function approximation by employing posterior sampling, which has been shown to empirically outperform the popular UCB algorithms in a wide range of regimes. We first introduce Delayed-PSVI, an optimistic value-based algorithm that effectively explores the value function space via noise perturbation with posterior sampling. We provide the first analysis for posterior sampling algorithms with delayed feedback in RL and show our algorithm achieves $\widetilde{O}(\sqrt{d^3H^3 T} + d^2H^2 E[τ])$ worst-case regret in the presence of unknown stochastic delays. Here $E[τ]$ is the expected delay. To further improve its computational efficiency and to expand its applicability in high-dimensional RL problems, we incorporate a gradient-based approximate sampling scheme via Langevin dynamics for Delayed-LPSVI, which maintains the same order-optimal regret guarantee with $\widetilde{O}(dHK)$ computational cost. Empirical evaluations are performed to demonstrate the statistical and computational efficacy of our algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
