单计数器马尔可夫决策过程
One-Counter Markov Decision Processes
摘要:
研究了一类有限表示的可数状态单计数马尔可夫决策过程的中心分析问题的计算复杂性。OC-MDP等价于(离散时间)拟生灭过程(QBD)的受控扩展,QBD是排队论和应用概率中大量研究的随机模型。因此,它们可以被看作是经典随机模型的自然"对抗“版本。或者,它们也可以被视为经典单计数器自动机的自然概率/受控扩展。OC-MDP还支持(作为一个非常有限的特殊情况)最近研究的MDP模型称为“偿付能力游戏”,该模型模拟了规避风险的赌博场景。关于这些模型的基本计算问题包括“终止”问题和"限制“问题,例如:控制器是否有一个"策略”(或"策略“)来确保计数器(例如,可以计算队列中的作业数)几乎肯定(a.s.)会达到值0(空队列)?或者它会有无限的limsup值,a.s.?或者,它将在选定的终端状态下达到值0,a.s.?或者,如果这些不满足,计算所有策略的最大(上确界)概率。我们提供了新的上限和下限的复杂性,这样的问题。对于其中的一些,我们提出了一个多项式时间算法,而对于其他人,我们显示PSPACE-或BH-硬度,并给出了EXPTIME上限。我们的上界结合了MDP奖励模型理论、随机游走理论和各种自动机理论方法的技术。
英文摘要:
We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs are equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a stochastic model studied heavily in queueing theory and applied probability. They can thus be viewed as a natural ``adversarial'' version of a classic stochastic model. Alternatively, they can also be viewed as a natural probabilistic/controlled extension of classic one-counter automata. OC-MDPs also subsume (as a very restricted special case) a recently studied MDP model called ``solvency games'' that model a risk-averse gambling scenario. Basic computational questions about these models include ``termination'' questions and ``limit'' questions, such as the following: does the controller have a ``strategy'' (or ``policy'') to ensure that the counter (which may for example count the number of jobs in the queue) will hit value 0 (the empty queue) almost surely (a.s.)? Or that it will have infinite limsup value, a.s.? Or, that it will hit value 0 in selected terminal states, a.s.? Or, in case these are not satisfied a.s., compute the maximum (supremum) such probability over all strategies. We provide new upper and lower bounds on the complexity of such problems. For some of them we present a polynomial-time algorithm, whereas for others we show PSPACE- or BH-hardness and give an EXPTIME upper bound. Our upper bounds combine techniques from the theory of MDP reward models, the theory of random walks, and a variety of automata-theoretic methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。