用于增量自主探索的分层状态发现
Layered State Discovery for Incremental Autonomous Exploration
摘要:
我们研究了Lim & Auer(2012)提出的自主探索(AX)问题。在这种设置中,目标是发现一组$ε$ -最优策略,其达到一组$\mathcal{S}_L^{\rightarrow}$的递增式$L$ -可控状态。我们引入了一种新的分层分解的增量$L$可控状态的状态扩展算子的迭代应用程序的基础上。我们利用这些结果来设计分层自主探索(LAE),AX的一种新算法,达到$\tilde{\mathcal{O}}(LS^{\rightarrow}_{L(1+ε)}Γ_{L(1+ε)} A \ln^{12}(S^{\rightarrow}_{L(1+ε)})/ε^2)$的样本复杂度,其中$S^{\rightarrow}_{L(1+ε)}$是递增$L(1+ε)$可控的状态数,$A$是动作数,$Γ_{L(1+ε)}$是这些状态上的转换的分支因子。LAE通过$L^2$因子改进了Tarbouriech等人(2020 a)的算法,并且它是在可数无限状态空间中工作的AX的第一个算法。此外,我们证明了,在一定的可识别性假设下,LAE实现了$\tilde{\mathcal{O}}(LS^{\rightarrow}_{L}A\ln^{12}(S^{\rightarrow}_{L})/ε^2)$的最小最优样本复杂度,优于现有算法,并首次匹配Cai等人证明的下限。(2022)对数因子。
英文摘要:
We study the autonomous exploration (AX) problem proposed by Lim & Auer (2012). In this setting, the objective is to discover a set of $ε$-optimal policies reaching a set $\mathcal{S}_L^{\rightarrow}$ of incrementally $L$-controllable states. We introduce a novel layered decomposition of the set of incrementally $L$-controllable states that is based on the iterative application of a state-expansion operator. We leverage these results to design Layered Autonomous Exploration (LAE), a novel algorithm for AX that attains a sample complexity of $\tilde{\mathcal{O}}(LS^{\rightarrow}_{L(1+ε)}Γ_{L(1+ε)} A \ln^{12}(S^{\rightarrow}_{L(1+ε)})/ε^2)$, where $S^{\rightarrow}_{L(1+ε)}$ is the number of states that are incrementally $L(1+ε)$-controllable, $A$ is the number of actions, and $Γ_{L(1+ε)}$ is the branching factor of the transitions over such states. LAE improves over the algorithm of Tarbouriech et al. (2020a) by a factor of $L^2$ and it is the first algorithm for AX that works in a countably-infinite state space. Moreover, we show that, under a certain identifiability assumption, LAE achieves minimax-optimal sample complexity of $\tilde{\mathcal{O}}(LS^{\rightarrow}_{L}A\ln^{12}(S^{\rightarrow}_{L})/ε^2)$, outperforming existing algorithms and matching for the first time the lower bound proved by Cai et al. (2022) up to logarithmic factors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。