卷积投影：连续空间 MDP 中强化学习的最优样本复杂度
Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs
AI总结：
针对平滑连续空间 MDP，论文提出基于调和分析投影与正交三角多项式特征的扰动最小二乘值迭代，获得速率最优样本复杂度并统一 Lipschitz 与低秩 MDP 结果。
摘要：
我们研究在一类具有平滑 Bellman 算子的一般连续空间 Markov 决策过程（MDP）中学习 ε-最优策略的问题。在可访问生成模型的条件下，我们通过执行一种简单的、扰动版最小二乘值迭代，并以正交三角多项式作为特征，实现了速率最优的样本复杂度。我们解决方案的关键是一种源于调和分析思想的新型投影技术。我们的 \widetilde{\mathcal{O}}(ε^{-2-d/(ν+1)}) 样本复杂度（其中 d 是状态-动作空间的维度，ν 是平滑阶数）在 Lipschitz MDP（ν=0）这一特殊情形下恢复了离散化方法的当前最优结果。同时，当 ν→∞ 时，它恢复并大幅推广了更适合回归方法的低秩 MDP 的 \mathcal{O}(ε^{-2}) 速率。从这个意义上说，我们的结果弥合了连续空间 MDP 上两种流行但相互冲突的观点之间的差距。
英文摘要：
We consider the problem of learning an $\varepsilon$-optimal policy in a general class of continuous-space Markov decision processes (MDPs) having smooth Bellman operators. Given access to a generative model, we achieve rate-optimal sample complexity by performing a simple, \emph{perturbed} version of least-squares value iteration with orthogonal trigonometric polynomials as features. Key to our solution is a novel projection technique based on ideas from harmonic analysis. Our~$\widetilde{\mathcal{O}}(ε^{-2-d/(ν+1)})$ sample complexity, where $d$ is the dimension of the state-action space and $ν$ the order of smoothness, recovers the state-of-the-art result of discretization approaches for the special case of Lipschitz MDPs $(ν=0)$. At the same time, for $ν\to\infty$, it recovers and greatly generalizes the $\mathcal{O}(ε^{-2})$ rate of low-rank MDPs, which are more amenable to regression approaches. In this sense, our result bridges the gap between two popular but conflicting perspectives on continuous-space MDPs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
