分布失配引起的扰动复杂性：再生核Hilbert空间中强化学习的系统分析
Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space
摘要：
现有的强化学习（RL）理论分析大多局限于表格设置或线性模型，这是由于难以处理高维空间和不确定环境中的函数逼近问题。这项工作提供了一个新的视角到这一挑战，通过分析RL在一般的再生核希尔伯特空间（RKHS）。考虑一类Markov决策过程$\mathcal{M}$，其奖励函数位于RKHS的单位球内，转移概率位于给定的任意集合内.我们定义了一个称为扰动复杂度的量由分布失配$Δ_{\mathcal{M}}(ε)$来表征的复杂性，可容许的状态-动作分布空间中的RKHS与尺度$ε$的扰动。我们表明，$Δ_{\mathcal{M}}(ε)$给出了所有可能的算法的误差的下限和两个特定的算法（拟合奖励和拟合Q-迭代）的RL问题的上限。因此，$Δ_\mathcal{M}(ε)$相对于$ε$的衰减度量了RL问题在$\mathcal{M}$上的难度。我们还提供了一些具体的例子，并讨论了在这些例子中，$Δ_{\mathcal{M}}(ε)$是否衰减快。作为一个副产品，我们表明，当奖励函数位于一个高维RKHS，即使转移概率是已知的，行动空间是有限的，它仍然是可能的RL问题，从维数灾难。
英文摘要：
Most existing theoretical analysis of reinforcement learning (RL) is limited to the tabular setting or linear models due to the difficulty in dealing with function approximation in high dimensional space with an uncertain environment. This work offers a fresh perspective into this challenge by analyzing RL in a general reproducing kernel Hilbert space (RKHS). We consider a family of Markov decision processes $\mathcal{M}$ of which the reward functions lie in the unit ball of an RKHS and transition probabilities lie in a given arbitrary set. We define a quantity called perturbational complexity by distribution mismatch $Δ_{\mathcal{M}}(ε)$ to characterize the complexity of the admissible state-action distribution space in response to a perturbation in the RKHS with scale $ε$. We show that $Δ_{\mathcal{M}}(ε)$ gives both the lower bound of the error of all possible algorithms and the upper bound of two specific algorithms (fitted reward and fitted Q-iteration) for the RL problem. Hence, the decay of $Δ_\mathcal{M}(ε)$ with respect to $ε$ measures the difficulty of the RL problem on $\mathcal{M}$. We further provide some concrete examples and discuss whether $Δ_{\mathcal{M}}(ε)$ decays fast or not in these examples. As a byproduct, we show that when the reward functions lie in a high dimensional RKHS, even if the transition probability is known and the action space is finite, it is still possible for RL problems to suffer from the curse of dimensionality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
