马尔可夫决策过程的Gauss-Newton方法
A Gauss-Newton Method for Markov Decision Processes
摘要：
近似牛顿方法是一种标准的优化工具，其目的是保持牛顿方法的优点，例如快速收敛，同时减轻其缺点，例如计算昂贵的计算或逆Hessian的估计。在这项工作中，我们研究近似牛顿方法的政策优化马尔可夫决策过程（MDP）。我们首先分析了MDP目标函数的Hessian结构。我们表明，像梯度，海森表现出有用的结构的背景下，MDPs和我们使用这种分析，以激励两个高斯-牛顿方法MDPs。与非线性最小二乘法的高斯-牛顿法一样，这些方法涉及通过忽略海森函数中难以估计的某些项来近似海森函数。近似Hessians具有理想的性能，如负定性，我们证明了几个重要的性能保证，包括保证上升方向，仿射变换的参数空间的不变性，收敛保证。最后，我们提供了一个统一的角度来看，关键的政策搜索算法，证明我们的第二个高斯-牛顿算法是密切相关的EM算法和自然梯度上升应用于MDP，但在实践中表现得更好的一系列具有挑战性的领域。
英文摘要：
Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton methods for policy optimization in Markov Decision Processes (MDPs). We first analyse the structure of the Hessian of the objective function for MDPs. We show that, like the gradient, the Hessian exhibits useful structure in the context of MDPs and we use this analysis to motivate two Gauss-Newton Methods for MDPs. Like the Gauss-Newton method for non-linear least squares, these methods involve approximating the Hessian by ignoring certain terms in the Hessian which are difficult to estimate. The approximate Hessians possess desirable properties, such as negative definiteness, and we demonstrate several important performance guarantees including guaranteed ascent directions, invariance to affine transformation of the parameter space, and convergence guarantees. We finally provide a unifying perspective of key policy search algorithms, demonstrating that our second Gauss-Newton algorithm is closely related to both the EM-algorithm and natural gradient ascent applied to MDPs, but performs significantly better in practice on a range of challenging domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
