arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1410.7583cs.CCcs.DM
收藏

策略迭代最坏情况复杂度的改进界

Improved bound on the worst case complexity of Policy Iteration

Romain Hollanders, Balázs Gerencsér, Jean-Charles Delvenne, Raphaël M. Jungers

摘要:

求解马尔可夫决策过程(MDP)是工程中经常出现的任务。尽管已知可以使用线性规划技术在多项式时间内找到用于最小化无限时域期望回报的解决方案,但是像策略迭代算法(PI)的迭代法在实践中通常仍然是最有效的。该方法保证在有限步内收敛。不幸的是,它是已知的,它可能需要一个指数数量的步骤的大小的问题收敛。另一方面,考虑到实际最坏情况的复杂性,许多悬而未决的问题仍然存在。在这项工作中,我们首次改进了Mansour & Singh(1999)提出的15年前的上界,证明PI最多需要k/(k-1)*k^n/n + o(k^n/n)次迭代才能收敛,其中n是MDP的状态数,k是每个状态的最大动作数。也许更重要的是,我们还表明,这个界是最佳的一个重要的放松的问题。

英文摘要:

Solving Markov Decision Processes (MDPs) is a recurrent task in engineering. Even though it is known that solutions for minimizing the infinite horizon expected reward can be found in polynomial time using Linear Programming techniques, iterative methods like the Policy Iteration algorithm (PI) remain usually the most efficient in practice. This method is guaranteed to converge in a finite number of steps. Unfortunately, it is known that it may require an exponential number of steps in the size of the problem to converge. On the other hand, many open questions remain considering the actual worst case complexity. In this work, we provide the first improvement over the fifteen years old upper bound from Mansour & Singh (1999) by showing that PI requires at most k/(k-1)*k^n/n + o(k^n/n) iterations to converge, where n is the number of states of the MDP and k is the maximum number of actions per state. Perhaps more importantly, we also show that this bound is optimal for an important relaxation of the problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑