arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1306.0386math.OCcs.AIcs.DMcs.RO
收藏

策略迭代复杂度的改进和推广的上界

Improved and Generalized Upper Bounds on the Complexity of Policy Iteration

INRIA Nancy Grand Est(法国国家信息与自动化研究所南锡-大东部研究中心)

Bruno Scherrer

摘要:

给定具有$n$状态和动作总数$m$的马尔可夫决策过程(MDP),我们研究策略迭代(PI)算法收敛到最佳$γ$折扣策略所需的迭代次数。我们考虑PI的两个变化:霍华德的PI,改变了所有国家的行动与积极的优势,和单纯形PI,只改变了国家的行动与最大的优势。我们表明,霍华德的PI终止后,在最多$O\left(\frac{m}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$迭代,提高了一个因子$O(\log n)$的结果汉森等人,而Simplex-PI在最多$O\left(\frac{nm}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$迭代之后终止,将结果提高因子$O(\log n)$ Ye。在MDP的一些结构性质下,我们考虑了与折扣因子$γ$无关的界:假设过程从均匀分布开始,则兴趣量分别是\emph{expected time spent in transientstates}和\emph{the inverse of the frequency of visits in recurrentstates}上的界$τ\_t$和$τ\_r$-在所有状态和策略上是均匀的。这推广了Post & Ye关于确定性MDP的一个结果,其中$τ\_t\le 1$和$τ\_r \le n$,特别是它表明了Simplex-PI对于更大类的MDP是强多项式的.我们解释了为什么类似的结果似乎很难得出霍华德的PI。最后,根据额外的(限制性)假设,状态空间被划分为两组,分别是瞬态和经常性的所有政策的状态,我们表明,霍华德PI和单纯PI终止后,在最多$\tilde O(m(n^2τ\_t+nτ\_r))$迭代。

英文摘要:

Given a Markov Decision Process (MDP) with $n$ states and a totalnumber $m$ of actions, we study the number of iterations needed byPolicy Iteration (PI) algorithms to converge to the optimal$γ$-discounted policy. We consider two variations of PI: Howard'sPI that changes the actions in all states with a positive advantage,and Simplex-PI that only changes the action in the state with maximaladvantage. We show that Howard's PI terminates after at most $O\left(\frac{m}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$iterations, improving by a factor $O(\log n)$ a result by Hansen etal., while Simplex-PI terminates after at most $O\left(\frac{nm}{1-γ}\log\left(\frac{1}{1-γ}\right)\right)$iterations, improving by a factor $O(\log n)$ a result by Ye. Undersome structural properties of the MDP, we then consider bounds thatare independent of the discount factor~$γ$: quantities ofinterest are bounds $τ\_t$ and $τ\_r$---uniform on all states andpolicies---respectively on the \emph{expected time spent in transientstates} and \emph{the inverse of the frequency of visits in recurrentstates} given that the process starts from the uniform distribution.Indeed, we show that Simplex-PI terminates after at most $\tilde O\left(n^3 m^2 τ\_t τ\_r \right)$ iterations. This extends arecent result for deterministic MDPs by Post & Ye, in which $τ\_t\le 1$ and $τ\_r \le n$, in particular it shows that Simplex-PI isstrongly polynomial for a much larger class of MDPs. We explain whysimilar results seem hard to derive for Howard's PI. Finally, underthe additional (restrictive) assumption that the state space ispartitioned in two sets, respectively states that are transient andrecurrent for all policies, we show that both Howard's PI andSimplex-PI terminate after at most $\tilde O(m(n^2τ\_t+nτ\_r))$iterations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑