arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1104.3348cs.GT
收藏

Büchi目标马氏决策过程定性分析的符号算法

Symbolic Algorithms for Qualitative Analysis of Markov Decision Processes with Büchi Objectives

Krishnendu Chatterjee, Monika Henzinger, Manas Joglekar, Nisarg Shah

摘要:

我们考虑具有ω-正则规格的马尔可夫决策过程(MDP)作为奇偶目标.我们考虑的问题,计算一组几乎肯定获胜的状态,从那里可以确保目标的概率为1。用于计算奇偶目标的几乎确定获胜集的算法迭代地使用Büchi目标(奇偶目标的特殊情况)的几乎确定获胜集的解。我们的贡献如下:首先,我们提出了第一个次二次符号算法来计算Büchi目标的MDP的几乎肯定获胜集;我们的算法需要O(n \sqrt{m})的符号步骤相比,以前已知的算法,需要O(n^2)的符号步骤,其中$n$是状态的数量和$m$是MDP的边缘的数量。在实践中,MDP有恒定的出度,然后我们的符号算法需要O(n \sqrt{n})的符号步骤,相比以前已知的$O(n^2)$符号步骤算法。其次,我们提出了一个新的算法,即Win-Lose算法,它具有以下两个性质:(a)与以往的算法相比,该算法迭代地计算几乎必然获胜集及其补集的子集,而以往的算法都是在终止时发现几乎必然获胜集;和(b)需要O(n \sqrt{K})个符号步骤,其中K是MDP的强连通分量(SCC's)的最大边数。输赢算法需要对SCC进行符号计算。第三,我们改进了符号SCC计算的算法;以前已知的算法需要线性符号步骤,我们的新算法改进了与线性步骤数相关的常数。在最坏的情况下,以前已知的算法需要5 n个符号步骤,而我们的新算法需要4 n个符号步骤。

英文摘要:

We consider Markov decision processes (MDPs) with ω-regular specifications given as parity objectives. We consider the problem of computing the set of almost-sure winning states from where the objective can be ensured with probability 1. The algorithms for the computation of the almost-sure winning set for parity objectives iteratively use the solutions for the almost-sure winning set for Büchi objectives (a special case of parity objectives). Our contributions are as follows: First, we present the first subquadratic symbolic algorithm to compute the almost-sure winning set for MDPs with Büchi objectives; our algorithm takes O(n \sqrt{m}) symbolic steps as compared to the previous known algorithm that takes O(n^2) symbolic steps, where $n$ is the number of states and $m$ is the number of edges of the MDP. In practice MDPs have constant out-degree, and then our symbolic algorithm takes O(n \sqrt{n}) symbolic steps, as compared to the previous known $O(n^2)$ symbolic steps algorithm. Second, we present a new algorithm, namely win-lose algorithm, with the following two properties: (a) the algorithm iteratively computes subsets of the almost-sure winning set and its complement, as compared to all previous algorithms that discover the almost-sure winning set upon termination; and (b) requires O(n \sqrt{K}) symbolic steps, where K is the maximal number of edges of strongly connected components (scc's) of the MDP. The win-lose algorithm requires symbolic computation of scc's. Third, we improve the algorithm for symbolic scc computation; the previous known algorithm takes linear symbolic steps, and our new algorithm improves the constants associated with the linear number of steps. In the worst case the previous known algorithm takes 5n symbolic steps, whereas our new algorithm takes 4n symbolic steps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑