对Fisher指数平均法的再思考
Rethinking Exponential Averaging of the Fisher
摘要：
在机器学习（ML）的优化中，曲率矩阵（CM）估计通常依赖于局部估计的指数平均（EA）（给出EA-CM算法）。这一方法在原则上没有什么道理，但在实践中经常使用。在本文中，我们绘制了EA-CM算法和我们所谓的“二次正则化模型的唤醒”之间的连接。概述的连接使我们能够从优化的角度理解EA-CM算法正在做什么。从建立的连接，我们提出了一个新的家庭的算法，“KL发散唤醒正则化模型”（KLD-WRM）。我们给出了三个不同的KLD-WRM的实际实例，并在数值上表明，这些优于K-FAC MNIST。
英文摘要：
In optimization for Machine learning (ML), it is typical that curvature-matrix (CM) estimates rely on an exponential average (EA) of local estimates (giving EA-CM algorithms). This approach has little principled justification, but is very often used in practice. In this paper, we draw a connection between EA-CM algorithms and what we call a "Wake of Quadratic regularized models". The outlined connection allows us to understand what EA-CM algorithms are doing from an optimization perspective. Generalizing from the established connection, we propose a new family of algorithms, "KL-Divergence Wake-Regularized Models" (KLD-WRM). We give three different practical instantiations of KLD-WRM, and show numerically that these outperform K-FAC on MNIST.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
