一般函数逼近下的非平稳强化学习
Non-stationary Reinforcement Learning under General Function Approximation
AI总结：
针对一般函数逼近下的非平稳 MDP，本文提出 DBE 维数与带滑动窗口的无模型算法 SW-OPEA，建立动态遗憾界并证明其优于现有 UCB 型方法。
摘要：
一般函数逼近是在广泛的强化学习（RL）场景中处理巨大状态和动作空间的有力工具。然而，对具有一般函数逼近的非平稳 MDP 的理论理解仍然有限。本文中，我们首次进行了此类尝试。我们首先针对非平稳 MDP 提出了一种称为动态 Bellman Eluder（DBE）维数的新复杂度度量，它涵盖了静态 MDP 以及非平稳 MDP 中大多数现有可处理 RL 问题。基于所提出的复杂度度量，我们提出了一种名为 SW-OPEA 的新型基于置信集的无模型算法，其特点是滑动窗口机制和针对非平稳 MDP 的新置信集设计。随后，我们为所提算法建立了动态遗憾的上界，并表明只要变化预算不特别大，SW-OPEA 就具有可证明的效率。我们进一步通过非平稳线性 MDP 和表格 MDP 的例子证明，在小变化预算情形下，我们的算法比现有 UCB 型算法表现更好。据我们所知，这是在具有一般函数逼近的非平稳 MDP 中首次进行动态遗憾分析。
英文摘要：
General function approximation is a powerful tool to handle large state and action spaces in a broad range of reinforcement learning (RL) scenarios. However, theoretical understanding of non-stationary MDPs with general function approximation is still limited. In this paper, we make the first such an attempt. We first propose a new complexity metric called dynamic Bellman Eluder (DBE) dimension for non-stationary MDPs, which subsumes majority of existing tractable RL problems in static MDPs as well as non-stationary MDPs. Based on the proposed complexity metric, we propose a novel confidence-set based model-free algorithm called SW-OPEA, which features a sliding window mechanism and a new confidence set design for non-stationary MDPs. We then establish an upper bound on the dynamic regret for the proposed algorithm, and show that SW-OPEA is provably efficient as long as the variation budget is not significantly large. We further demonstrate via examples of non-stationary linear and tabular MDPs that our algorithm performs better in small variation budget scenario than the existing UCB-type algorithms. To the best of our knowledge, this is the first dynamic regret analysis in non-stationary MDPs with general function approximation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
