马尔可夫决策过程的基于分布的目标
Distribution-based objectives for Markov Decision Processes
摘要：
我们考虑马尔可夫决策过程（MDP）的基于分布的目标。这类目标在全部信息和部分信息之间产生了一个有趣的权衡。与完全观察一样，MDP中的策略可以依赖于系统的状态，但与部分信息类似，策略需要同时考虑所有状态。在本文中，我们专注于两个安全问题，自然出现在这种情况下，即存在和普遍的安全。给定一个MDP A和一个A状态上概率分布的闭凸多面体H，存在安全问题询问H中是否存在某种分布d和A的策略，使得从d开始并重复应用该策略，该分布永远保持在H中。普适安全问题问的是，对于H中的所有分布，是否存在这样一种策略A，使该分布永远保持在H中。我们证明了这两个问题都是可判定的，具有严格的复杂性界限：我们证明了存在安全是PTIME完全的，而普遍安全是co-NP完全的。此外，我们将这些结果与拉宾的概率有限状态自动机（PFA）的存在性和普遍的安全问题进行比较，PFA是部分可观测MDP的子类，具有零观测。与MDPs相比，PFAs的策略不依赖于状态。与PTIME结果形成鲜明对比的是，我们发现PFAs的存在安全性是不可判定的，H具有封闭和开放的边界。另一方面，证明了PFAs的普遍安全性在EXPTIME中是可判定的，具有co-NP下界。最后，我们表明，输入多面体的替代表示使我们能够提高MDP和PFAs的通用安全性的复杂性。
英文摘要：
We consider distribution-based objectives for Markov Decision Processes (MDP). This class of objectives gives rise to an interesting trade-off between full and partial information. As in full observation, the strategy in the MDP can depend on the state of the system, but similar to partial information, the strategy needs to account for all the states at the same time. In this paper, we focus on two safety problems that arise naturally in this context, namely, existential and universal safety. Given an MDP A and a closed and convex polytope H of probability distributions over the states of A, the existential safety problem asks whether there exists some distribution d in H and a strategy of A, such that starting from d and repeatedly applying this strategy keeps the distribution forever in H. The universal safety problem asks whether for all distributions in H, there exists such a strategy of A which keeps the distribution forever in H. We prove that both problems are decidable, with tight complexity bounds: we show that existential safety is PTIME-complete, while universal safety is co-NP-complete. Further, we compare these results with existential and universal safety problems for Rabin's probabilistic finite-state automata (PFA), the subclass of Partially Observable MDPs which have zero observation. Compared to MDPs, strategies of PFAs are not state-dependent. In sharp contrast to the PTIME result, we show that existential safety for PFAs is undecidable, with H having closed and open boundaries. On the other hand, it turns out that the universal safety for PFAs is decidable in EXPTIME, with a co-NP lower bound. Finally, we show that an alternate representation of the input polytope allows us to improve the complexity of universal safety for MDPs and PFAs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
