arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03048math.OCmath.AGstat.ML
收藏

部分可观测马尔可夫决策过程中的值函数半代数集

The Value Function Semi-Algebraic Set in Partially Observable Markov Decision Processes

University of California, Los Angeles(加州大学洛杉矶分校);Max Planck Institute for Mathematics in the Sciences(马克斯·普朗克科学促进会科学数学研究所)

Ryan A. Anderson, Guido Montufar

AI总结:

本文刻画了无限时域部分可观测马尔可夫决策过程(POMDP)中无记忆随机策略下可行值函数的几何结构,将其表示为由多项式不等式定义的半代数集,并揭示了部分可观测性导致的非线性约束和孤立局部极大值现象。

摘要:

我们研究了无限时域部分可观测马尔可夫决策过程(POMDP)中无记忆随机策略下可行值函数的几何结构。我们的主要贡献是将可行值函数集刻画为一个半代数集,由POMDP的转移动力学、观测核和奖励结构决定的显式多项式不等式定义。这一结果将先前完全可观测马尔可夫决策过程(其中可行集已知为多面体)的工作扩展到了更为复杂的部分可观测情形。与MDP中出现的多面体结构不同,部分可观测性引入了本质非线性的约束,导致更丰富和更复杂的几何结构。我们的几何刻画为MDP和POMDP中策略优化的景观提供了新的见解,并揭示了部分可观测性独有的定性现象,包括长期奖励的孤立局部极大值及其对初始状态分布的依赖性。

英文摘要:

We study the geometry of feasible value functions in infinite-horizon partially observable Markov decision processes (POMDPs) under memoryless stochastic policies. Our main contribution is a characterization of the feasible set of value functions as a semi-algebraic set, defined by explicit polynomial inequalities determined by the transition dynamics, observation kernel, and reward structure of the POMDP. This result extends prior work for fully observable Markov decision processes, where the feasible set is known to be a polytope, to the substantially more intricate partially observable setting. In contrast to the polyhedral structure arising in MDPs, partial observability induces fundamentally nonlinear constraints, leading to a richer and more complex geometric structure. Our geometric characterization provides new insight into the landscape of policy optimization in both MDPs and POMDPs, and reveals qualitative phenomena unique to partial observability, including the emergence of isolated local maximizers of the long-term reward and their dependence on the initial state distribution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑