一类精确可解模型的策略梯度研究
A Study of Policy Gradient on a Class of Exactly Solvable Models
摘要：
策略梯度方法被广泛用于强化学习中，作为优化期望收益的一种方法。在本文中，我们探讨的政策参数的演变，为一类特殊的精确可解POMDPs，作为一个连续状态马尔可夫链，其转移概率是由政策的值的分布的梯度。我们的方法在很大程度上依赖于随机游走理论，特别是仿射Weyl群。我们构造了一类新的部分可观测的环境，可控的探索难度，在其中的价值分布，因此，政策参数的演变，可以推导出解析。利用这些环境，我们分析了策略梯度的概率收敛到不同的局部最大值的价值函数。据我们所知，这是第一种方法，分析计算景观的策略梯度POMDPs一类这样的环境，导致有趣的见解，这个问题的难度。
英文摘要：
Policy gradient methods are extensively used in reinforcement learning as a way to optimize expected return. In this paper, we explore the evolution of the policy parameters, for a special class of exactly solvable POMDPs, as a continuous-state Markov chain, whose transition probabilities are determined by the gradient of the distribution of the policy's value. Our approach relies heavily on random walk theory, specifically on affine Weyl groups. We construct a class of novel partially observable environments with controllable exploration difficulty, in which the value distribution, and hence the policy parameter evolution, can be derived analytically. Using these environments, we analyze the probabilistic convergence of policy gradient to different local maxima of the value function. To our knowledge, this is the first approach developed to analytically compute the landscape of policy gradient in POMDPs for a class of such environments, leading to interesting insights into the difficulty of this problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
