连续空间POMDP的近优近似与有限记忆策略
Near Optimal Approximations and Finite Memory Policies for POMPDs with Continuous Spaces
AI总结:
针对连续空间POMDP的近似求解问题,提出基于观测空间离散化与有限历史信息的严格近似方法,证明策略近优性,还提出有限记忆Q学习算法并证明其收敛性。
摘要:
我们研究连续空间部分可观测马尔可夫决策过程(partially observed Markov decision processes, POMDPs)的近似方法。作为研究POMDP的标准方法,信念MDP约简(Belief MDP reduction)因状态空间被提升至概率测度空间,实际应用需要严格的近似方法。本文推广近期研究成果,通过离散化观测空间,并利用离散观测与控制动作的有限长度历史构建完全可观测有限MDP模型,提出了严格的近似方法。我们证明,在信道的某些正则性假设以及隐状态过程的特定受控滤波器稳定性要求下,所得策略是近优的。此外,通过对测量值进行量化,我们能够采用更精细的滤波器稳定性条件。我们还提出了一种利用离散化信息变量有限记忆的Q学习算法,并证明其收敛至采用该近似方法构建的有限完全可观测MDP的最优性方程。
英文摘要:
We study an approximation method for partially observed Markov decision processes (POMDPs) with continuous spaces. Belief MDP reduction, which has been the standard approach to study POMDPs requires rigorous approximation methods for practical applications, due to the state space being lifted to the space of probability measures. Generalizing recent work, in this paper we present rigorous approximation methods via discretizing the observation space and constructing a fully observed finite MDP model using a finite length history of the discrete observations and control actions. We show that the resulting policy is near-optimal under some regularity assumptions on the channel, and under certain controlled filter stability requirements for the hidden state process. Furthermore, by quantizing the measurements, we are able to utilize refined filter stability conditions. We also provide a Q learning algorithm that uses a finite memory of discretized information variables, and prove its convergence to the optimality equation of the finite fully observed MDP constructed using the approximation method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。