arXiv 1202.62592012-02-28math.OC

概率空间的距离，以及马尔可夫决策过程和重复博弈中的长期值

A distance for probability spaces, and long-term values in Markov Decision Processes and Repeated Games

Jérôme Renault, Xavier Venel

摘要：

给定一个有限集$K$，我们用$X=Δ(K)$表示$K$上的概率集，用$Z=Δ_f(X)$表示$X$上具有有限支撑的Borel概率集。研究一个在$K$上具有部分信息的马尔可夫决策过程，自然会得到一个在$X$上具有全部信息的马尔可夫决策过程。我们在$Z$上引入了一个新的度量$d_*$，使得从$(X, \|.\|_1)$到$(Z,d_*)$的转换成为1-Lipschitz.在文章的第一部分中，我们定义并证明了度量$d_*$的几个性质。特别地，$d_*$满足Kantorovich-Rubinstein型对偶公式，并且可以用衰变来刻画。在第二部分中，我们刻画了几类紧非扩张马氏决策过程的极限值。特别是，我们使用的度量$d_*$的特点，在部分观察MDP的极限值与许多国家和重复游戏的通知控制器有限的状态和行动。此外，在每一种情况下，我们可以证明存在一个广义的概念，一致的价值，我们认为不仅是塞萨尔平均数时的阶段是足够大，但任何评价功能$θ\in Δ(\N^*)$时，不耐烦$I(θ)=\sum_{t\geq 1} |θ_{t+1}-θ_t|$是足够小的。

英文摘要：

Given a finite set $K$, we denote by $X=Δ(K)$ the set of probabilities on $K$ and by $Z=Δ_f(X)$ the set of Borel probabilities on $X$ with finite support. Studying a Markov Decision Process with partial information on $K$ naturally leads to a Markov Decision Process with full information on $X$. We introduce a new metric $d_*$ on $Z$ such that the transitions become 1-Lipschitz from $(X, \|.\|_1)$ to $(Z,d_*)$. In the first part of the article, we define and prove several properties of the metric $d_*$. Especially, $d_*$ satisfies a Kantorovich-Rubinstein type duality formula and can be characterized by using disintegrations. In the second part, we characterize the limit values in several classes of "compact non expansive" Markov Decision Processes. In particular we use the metric $d_*$ to characterize the limit value in Partial Observation MDP with finitely many states and in Repeated Games with an informed controller with finite sets of states and actions. Moreover in each case we can prove the existence of a generalized notion of uniform value where we consider not only the Cesàro mean when the number of stages is large enough but any evaluation function $θ\in Δ(\N^*)$ when the impatience $I(θ)=\sum_{t\geq 1} |θ_{t+1}-θ_t|$ is small enough.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文