arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19448math.OCcs.LGcs.NAcs.SYeess.SYmath.NAstat.ML
收藏

线性规划的Fisher-Rao梯度流与状态-动作自然策略梯度

Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients

RWTH Aachen University(亚琛工业大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Max Planck Institute for Mathematics in the Sciences(马克斯·普朗克数学科学研究所)

Johannes Müller, Semih Çaycı, Guido Montúfar

AI总结:

本文研究线性规划的Fisher-Rao梯度流,证明其线性收敛并改进熵正则化误差界,进而扩展至扰动情形,涵盖状态-动作自然策略梯度的次线性收敛分析。

摘要:

Kakade的自然策略梯度方法近年来被广泛研究,证明了在有正则化和无正则化情况下的线性收敛性。我们研究了另一种基于状态-动作分布的Fisher信息矩阵的自然梯度方法,该方法在理论方面受到的关注较少。在此,状态-动作分布在状态-动作多胞体内相对于线性势遵循Fisher-Rao梯度流。因此,我们更一般地研究线性规划的Fisher-Rao梯度流,并证明了线性收敛性,其收敛速率取决于线性规划的几何结构。等价地,这给出了线性规划熵正则化所引入误差的估计,改进了现有结果。我们扩展了这些结果,并证明了在存在近似误差的情况下,扰动Fisher-Rao梯度流和自然梯度流具有次线性收敛性。特别地,这些一般性结果涵盖了状态-动作自然策略梯度的情况。

英文摘要:

Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑