arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.09753cs.LGmath.OCstat.ML
收藏

部分可观测马尔可夫决策过程(POMDPs)下自然Actor-Critic的有限时间分析

Finite-Time Analysis of Natural Actor-Critic for POMDPs

RWTH Aachen University(亚琛工业大学);ETH Zurich(苏黎世联邦理工学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Semih Cayci, Niao He, R. Srikant

AI总结:

该研究针对大状态空间部分可观测马尔可夫决策过程的强化学习问题,分析带有限内部记忆与多步时序差分评估的自然actor-critic方法,首次证明其非渐近全局收敛性,刻画有限状态控制器带来的额外误差并给出减小该误差的途径。

摘要:

我们研究状态空间规模庞大甚至可数无穷的部分可观测马尔可夫决策过程(POMDPs)的强化学习问题,其中控制器仅能获取底层受控马尔可夫链的带噪声观测。我们采用一种自然actor-critic方法,该方法使用有限内部记忆进行策略参数化,并采用多步时序差分学习算法完成策略评估。据我们所知,我们首次证明了函数近似下部分可观测系统中actor-critic方法的非渐近全局收敛性。具体而言,除了马尔可夫决策过程(MDPs)中同样存在的函数近似误差与统计误差外,我们还明确刻画了因使用有限状态控制器而产生的误差。该额外误差通过POMDPs中传统信念状态与使用有限状态控制器时隐状态后验分布之间的全变差距离来表示。此外,我们证明在滑动块控制器的场景下,通过使用更大的块尺寸可使该误差变小。

英文摘要:

We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of the underlying controlled Markov chain. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization, and a multi-step temporal difference learning algorithm for policy evaluation. We establish, to the best of our knowledge, the first non-asymptotic global convergence of actor-critic methods for partially observed systems under function approximation. In particular, in addition to the function approximation and statistical errors that also arise in MDPs, we explicitly characterize the error due to the use of finite-state controllers. This additional error is stated in terms of the total variation distance between the traditional belief state in POMDPs and the posterior distribution of the hidden state when using a finite-state controller. Further, we show that this error can be made small in the case of sliding-block controllers by using larger block sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑