arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.13405stat.MLcs.LG
收藏

马尔可夫决策过程中最佳策略识别的自适应抽样

Adaptive Sampling for Best Policy Identification in Markov Decision Processes

UMPA, ENS Lyon(里昂高等师范学校 UMPA);KTH, Royal Institute of Technology(瑞典皇家理工学院)

Aymen Al Marjani, Alexandre Proutiere

摘要:

我们研究了折扣马尔可夫决策过程(MDP)的最佳策略识别问题时,学习者有机会获得一个生成模型。我们的目标是设计一个学习算法返回的最佳政策,尽可能早。我们首先推导出一个特定于问题的样本复杂度的下界满足任何学习算法。这个下限对应于一个最佳的样本分配,解决了一个非凸的程序,因此,很难利用在设计有效的算法。然后,我们提供了一个简单的和严格的样本复杂度下限的上限,其相应的接近最优的样本分配变得明确。上界取决于MDP的特定泛函,例如次优间隙和下一状态值函数的方差,从而真正捕获MDP的硬度。最后,我们设计KLB-TS(KL球跟踪和停止),跟踪这个接近最优的分配算法,并提供渐近保证其样本复杂性(几乎肯定和预期)。KLB-TS对国家的最先进的算法的优势进行了讨论和数值说明。

英文摘要:

We investigate the problem of best-policy identification in discounted Markov Decision Processes (MDPs) when the learner has access to a generative model. The objective is to devise a learning algorithm returning the best policy as early as possible. We first derive a problem-specific lower bound of the sample complexity satisfied by any learning algorithm. This lower bound corresponds to an optimal sample allocation that solves a non-convex program, and hence, is hard to exploit in the design of efficient algorithms. We then provide a simple and tight upper bound of the sample complexity lower bound, whose corresponding nearly-optimal sample allocation becomes explicit. The upper bound depends on specific functionals of the MDP such as the sub-optimality gaps and the variance of the next-state value function, and thus really captures the hardness of the MDP. Finally, we devise KLB-TS (KL Ball Track-and-Stop), an algorithm tracking this nearly-optimal allocation, and provide asymptotic guarantees for its sample complexity (both almost surely and in expectation). The advantages of KLB-TS against state-of-the-art algorithms are discussed and illustrated numerically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑