Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients
RWTH Aachen University(亚琛工业大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Max Planck Institute for Mathematics in the Sciences(马克斯·普朗克数学科学研究所)
Johannes Müller, Semih Çaycı, Guido Montúfar
备注:
25 pages, 4 figures, to appear at SIAM Journal on Optimization
发表信息:
SIAM Journal on Optimization, Vol. 35, Iss. 2 (2025), 10.1137/24M1653422
Kakade's natural policy gradient method has been studied extensively in recent years, showing linear convergence with and without regularization. We study another natural gradient method based on the Fisher information matrix of the state-action distributions which has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients.
Center for Advanced Intelligence Project, RIKEN AIP, Japan(日本RIKEN AIP高级智能项目中心);School of Mathematics, University of Edinburgh, UK(爱丁堡大学数学学院);School of Mathematics, University of Edinburgh, UK, The Alan Turing Institute, UK(爱丁堡大学数学学院、英国艾伦·图灵研究所)
The Advective Fisher-Rao Geometry of Deterministic Measure Transport
Institut für Mathematik, Technische Universität Berlin(柏林工业大学数学研究所);Max-Planck-Institut für Mathematik in den Naturwissenschaften(马克斯·普朗克数学研究所(自然科学领域))
A Fisher-Rao gradient flow for entropic mean-field min-max games
Heriot-Watt University(赫瑞瓦特大学);Maxwell Institute for Mathematical Sciences(麦克斯韦数学研究所);University of Edinburgh(爱丁堡大学);The Alan Turing Institute(阿兰·图灵研究所);Simtopia(Simtopia公司)