揭秘线性马尔可夫决策过程（MDPs）与新型动力学聚合框架
Demystifying Linear MDPs and Novel Dynamics Aggregation Framework
AI总结：
本研究证明线性MDP的特征维度下界为S/U，可能随状态空间规模增长，为此提出动力学聚合框架及带可证明保证的线性函数近似HRL算法，在分层结构环境中显著提升遗憾界性能。
摘要：
在本研究中，我们证明了在线性MDPs中，为了恰当地表示转移概率，特征维度$d$的下界为$S/U$，其中$S$为状态空间规模，$U$为直接可达状态的最大规模。因此，$d$仍可能随$S$按环境的直接可达性成比例增长。为解决线性MDPs的这一局限，我们提出了一种基于动力学的新型结构聚合框架，命名为“dynamics aggregation（动力学聚合）”。针对这一新提出的框架，我们设计了一种可证明高效的、采用线性函数近似的分层强化学习算法，该算法利用了聚合后的子结构。我们提出的算法具备统计效率，其遗憾值为$ \tilde{O} ( d_ψ^{3/2} H^{3/2}\sqrt{ N T} )$，其中$d_ψ$表示聚合后子MDP的特征维度，$N$表示聚合后子MDP的数量。我们证实，在大多数具有分层结构的真实环境中，$d_ψ^3 N \ll d^{3}$的条件很容易满足，这使得该算法的遗憾界相较于LSVI-UCB（其遗憾值为$ \tilde{O} (d^{3/2} H^{3/2} \sqrt{ T})$）有显著提升。据我们所知，本研究首次提出了具备可证明保证的线性函数近似分层强化学习（HRL）算法。
英文摘要：
In this work, we prove that, in linear MDPs, the feature dimension $d$ is lower bounded by $S/U$ in order to aptly represent transition probabilities, where $S$ is the size of the state space and $U$ is the maximum size of directly reachable states. Hence, $d$ can still scale with $S$ depending on the direct reachability of the environment. To address this limitation of linear MDPs, we propose a novel structural aggregation framework based on dynamics, named as the "dynamics aggregation". For this newly proposed framework, we design a provably efficient hierarchical reinforcement learning algorithm in linear function approximation that leverages aggregated sub-structures. Our proposed algorithm exhibits statistical efficiency, achieving a regret of $ \tilde{O} ( d_ψ^{3/2} H^{3/2}\sqrt{ N T} )$, where $d_ψ$ represents the feature dimension of aggregated subMDPs and $N$ signifies the number of aggregated subMDPs. We establish that the condition $d_ψ^3 N \ll d^{3}$ is readily met in most real-world environments with hierarchical structures, enabling a substantial improvement in the regret bound compared to LSVI-UCB, which enjoys a regret of $ \tilde{O} (d^{3/2} H^{3/2} \sqrt{ T})$. To the best of our knowledge, this work presents the first HRL algorithm with linear function approximation that offers provable guarantees.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
