将PageRank、时间反向和策略评估联系起来
Linking PageRank, Time Reversal, and Policy Evaluation
AI总结：
本文通过时间反向马尔可夫链建立策略评估与PageRank的联系，证明折扣因子对应于teleportation参数，奖励诱导重启分布，并推广到任意有限MDP。
摘要：
我们建立了马尔可夫决策过程中的策略评估与网络分析中的PageRank之间的联系。对于固定策略，我们显示，折扣马尔可夫决策过程的价值函数可通过合适定义的时间反向马尔可夫链的PageRank向量获得，除了显式的重新缩放。在此对应中，折扣因子扮演teleportation参数的角色，而奖励诱导重启分布。在不可约情况之外，通过引入准稳态分布和Doob h-变换，我们证明了一个通用分解定理，表明任意有限MDP的策略评估可归约到策略诱导马尔可夫链的复发和瞬态组件上的PageRank问题。该框架自然扩展到有终端状态的无折扣MDP和转移依赖奖励。我们通过一个在大确定性和随机图上粘性随机游走的数值例子，展示了本方法的效率。
英文摘要：
We establish a connection between policy evaluation in Markov decision processes and PageRank in network analysis. For a fixed policy, we show that the value function of a discounted Markov decision process can be obtained, up to an explicit rescaling, from the PageRank vector of a suitably defined time-reversed Markov chain. In this correspondence, the discount factor plays the role of the teleportation parameter, while rewards induce the restart distribution. Beyond the irreducible case, invoking quasi-stationary distributions and Doob $h$-transforms, we prove a general decomposition theorem showing that policy evaluation for arbitrary finite MDPs reduces to a collection of PageRank problems on the recurrent and transient components of the policy-induced Markov chain. This framework naturally extends to undiscounted MDPs with terminal states and to transition-dependent rewards. We conclude by showing efficiency of our approach on a numerical example of a sticky random walk on large deterministic and random graphs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
