arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10938cs.LO
收藏

具有先验和通用语义的多环境马尔可夫决策过程

Multi-Environment MDPs with Prior and Universal Semantics

Université Libre de Bruxelles(布鲁塞尔自由大学)

Benjamin Bordais, Jean-François Raskin

AI总结:

本文研究了多环境马尔可夫决策过程的先验和通用语义,提出了一种新的算法以计算先验价值,并证明通用价值等于先验价值的下确界,同时展示了先验 MEMDPs 是 POMDPs 的可计算子类。

摘要:

多环境马尔可夫决策过程(MEMDPs)通过为每个可能的环境提供一个概率转移函数,使状态可观察但环境不可知。先前的研究探讨了两种语义:(i)通用语义,其中对手选择环境;(ii)先验语义,其中环境在执行前从固定分布中选择。我们澄清了这两种语义之间的关系。对于偶数目标,我们证明了定性问题,即价值为一,是相同的,并开发了一种新的算法来计算具有先验语义的MEMDP的一般价值。特别地,我们证明了具有偶数目标的MEMDP的先验价值可以以任意精度近似,通过一个空间高效的算法;等价地,当概率以 unary 表示时,关联的间隙问题在 PSPACE 内可判定,否则在 EXPSPACE 内可判定。我们然后证明通用价值等于所有信念下的先验价值的下确界。这为通用间隙问题提供了一种新的算法,其复杂度相同(unary 概率时为 PSPACE,一般情况为 EXPSPACE),优于先前的双指数空间过程。最后,我们观察到在先验语义下,MEMDPs 形成 POMDPs 的一个重要可计算子类:我们的算法利用信念熵永不增加的事实,并证明任何具有此性质的 POMDP 都可以有效地转化为先验 MEMDP,表明先验 MEMDPs 捕捉了 POMDPs 的广泛且实际相关的子类。

英文摘要:

Multiple-environment Markov decision processes (MEMDPs) equip an MDP with several probabilistic transition functions (one per possible environment) so that the state is observable but the environment is not. Previous work studies two semantics: (i) the universal semantics, where an adversary picks the environment; and (ii) the prior semantics, where the environment is drawn once before execution from a fixed distribution. We clarify the relation between these semantics. For parity objectives, we show that the qualitative questions, i.e. value one, coincide, and we develop a new algorithm for the general value of MEMDP with prior semantics. In particular, we show that the prior value of an MEMDP with a parity objective can be approximated to any precision with a space efficient algorithm; equivalently, the associated gap problem is decidable in PSPACE when probabilities are given in unary (and in EXPSPACE otherwise). We then prove that the universal value equals the infimum of prior values over all beliefs. This yields a new algorithm for the universal gap problem with the same complexity (PSPACE for unary probabilities, EXPSPACE in general), improving on earlier doubly-exponential-space procedures. Finally, we observe that MEMDPs under the prior semantics form an important tractable subclass of POMDPs: our algorithms exploit the fact that belief entropy never increases, and we establish that any POMDP with this property reduces effectively to a prior-MEMDP, showing that prior-MEMDPs capture a broad and practically relevant subclass of POMDPs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑