可数MDP中的瞬变
Transience in Countable MDPs
摘要：
Transience的目标不是无限频繁地访问任何状态。虽然这在有限的马尔可夫决策过程（MDP）中是不可能的，但它可以在可数的无限过程中得到满足，例如，如果转换图是非循环的。我们证明了以下基本性质的瞬态在可数无限MDPs。1.存在一致的$ε$ -最优的MD策略（无记忆确定性）的瞬态，即使在无限分支的MDPs。2.瞬时性的最优策略不需要存在，即使MDP是连续分支。然而，如果存在最优策略，则也存在最优MD策略。3.如果MDP是普遍瞬态的（即，在所有策略下几乎肯定是瞬时的），那么许多其他目标的策略复杂度比一般的MDP要低。例如，在一个示例中，$ε$ -最优策略的安全和co-Büchi和最优策略的$\{0,1,2\}$ -奇偶校验（如果它们存在）可以选择MD，即使MDP是无限分支。
英文摘要：
The Transience objective is not to visit any state infinitely often. While this is not possible in finite Markov Decision Process (MDP), it can be satisfied in countably infinite ones, e.g., if the transition graph is acyclic. We prove the following fundamental properties of Transience in countably infinite MDPs. 1. There exist uniformly $ε$-optimal MD strategies (memoryless deterministic) for Transience, even in infinitely branching MDPs. 2. Optimal strategies for Transience need not exist, even if the MDP is finitely branching. However, if an optimal strategy exists then there is also an optimal MD strategy. 3. If an MDP is universally transient (i.e., almost surely transient under all strategies) then many other objectives have a lower strategy complexity than in general MDPs. E.g., $ε$-optimal strategies for Safety and co-Büchi and optimal strategies for $\{0,1,2\}$-Parity (where they exist) can be chosen MD, even if the MDP is infinitely branching.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
