具有期望停时(Expected Stopping Time)的随机过程
Stochastic Processes with Expected Stopping Time
AI总结:
研究将马尔可夫决策过程停时推广为按概率分布选择且期望为T的问题,证明其精确求解与Positivity问题相互可归约,并给出近似版本在马尔可夫链和MDPs中的指数级时空复杂度解法。
摘要:
马尔可夫链是随机动态系统的事实上的有限状态模型,而马尔可夫决策过程(MDPs)通过引入非确定性行为对马尔可夫链进行了扩展。给定一个 MDP 和状态上的奖励,一个经典的优化准则是最大期望总奖励,其中 MDP 在 T 步后停止,这可以通过简单的动态规划算法来计算。我们考虑该问题的一个自然推广,其中停时可以根据概率分布来选择,使得期望停时为 T,以优化期望总奖励。令人惊讶的是,我们确立了马尔可夫链的期望停时问题与 Positivity 问题(这与著名的 Skolem 问题相关)之间的相互可归约性,而为该问题建立可判定性或不可判定性都将是一个重大突破。鉴于精确问题的困难性,我们考虑了该问题的近似版本:我们证明了它对于马尔可夫链可在指数时间内求解,对于 MDPs 可在指数空间内求解。
英文摘要:
Markov chains are the de facto finite-state model for stochastic dynamical systems, and Markov decision processes (MDPs) extend Markov chains by incorporating non-deterministic behaviors. Given an MDP and rewards on states, a classical optimization criterion is the maximal expected total reward where the MDP stops after T steps, which can be computed by a simple dynamic programming algorithm. We consider a natural generalization of the problem where the stopping times can be chosen according to a probability distribution, such that the expected stopping time is T, to optimize the expected total reward. Quite surprisingly we establish inter-reducibility of the expected stopping-time problem for Markov chains with the Positivity problem (which is related to the well-known Skolem problem), for which establishing either decidability or undecidability would be a major breakthrough. Given the hardness of the exact problem, we consider the approximate version of the problem: we show that it can be solved in exponential time for Markov chains and in exponential space for MDPs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。