arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00474stat.MLcs.LGcs.NAmath.NA
收藏

通过最小外围商进行马尔可夫链的持久-暂态策略评估

Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

Yang Xu, Vaneet Aggarwal

AI总结:

本文研究了可能可约且周期性的有限马尔可夫链的固定策略评估问题,提出通过最小外围商去除非衰减模式,重构有限时间回报并获得稳定估计。

摘要:

我们研究了可能可约且周期性的有限马尔可夫链的固定策略评估问题。经典评估方法在处理增益和偏差分解时存在局限:增益仅记录不变的塞萨罗平均值,而持久相依赖行为被包含在偏差中与真正的瞬态效应一起。我们识别出转移矩阵P的真实外围不变子空间K(P)是这种歧义的根源。通过K(P)商化是去除所有非衰减模式的最小精确商,使剩余动态严格稳定。选择一个核为K(P)的 gauge 投影Π后,奖励可唯一分解为r = g_Π^* + (I-P)v_Π^*,其中g_Π^*是持久模式轮廓,v_Π^*是 gauge 固定的瞬态成分。与经典归一化增益和偏差的精确比较显示,新对重新分配了相同信息,使得所有持久模式都体现在g_Π^*中,而v_Π^*是瞬态的。这种分解重构了有限时间回报,恢复了状态平均奖励,允许瞬态成本解释,并在生成模型下产生稳定估计器。

英文摘要:

We study fixed-policy evaluation for finite Markov chains that may be reducible and periodic. Classical evaluation methods with gain and bias decomposition are not always diagnostic: the gain records only invariant Cesàro averages, while persistent phase-dependent behavior is absorbed into the bias together with genuinely transient effects. We identify the real peripheral invariant subspace $\mathcal{K}(P)$ of the transition matrix $P$ as the source of this ambiguity. Quotienting by $\mathcal{K}(P)$ is the minimal exact quotient that removes all non-decaying modes and makes the remaining dynamics strictly stable. After choosing a gauge projection $Π$ with kernel $\mathcal{K}(P)$, the reward admits a unique decomposition $r = g_Π^\star + (I-P)v_Π^\star$, where $g_Π^\star$ is a persistent regime profile and $v_Π^\star$ is a gauge-fixed transient component. An exact comparison with classical normalized gain and bias shows that the new pair reallocates the same information into a persistent profile and a gauge-fixed representative of the transient quotient. This decomposition reconstructs finite-horizon returns, recovers statewise average reward, admits a transient-cost interpretation, and yields a stable estimator under a generative model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑