学习动力学的几何：优化山脊上的梯度下降与自然梯度
Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization
浏览论文内容
中文总结 AI 辅助
本文通过几何分析比较梯度下降与自然梯度下降在KLR训练的Hopfield网络优化山脊上的学习路径，发现自然梯度遵循测地路径、克服不稳定并更快收敛且泛化更优，揭示了优化动力学与表征几何的关联。
中文摘要 AI 辅助
基于核逻辑回归（KLR）的高容量联想记忆表现出一种“优化山脊”，其特征是极端的稳定性和高度偏斜的权重谱。然而，学习收敛到这一临界状态的动力学过程此前尚不清楚。本文对KLR训练的Hopfield网络的统计流形上的学习轨迹进行了几何分析。通过比较梯度下降（GD）和自然梯度下降（NGD）的路径，我们阐明了控制优化过程的机制。我们的分析揭示，山脊上的学习分两个不同阶段进行。我们表明，山脊的极端曲率导致标准GD遵循高度振荡、非测地的路径。与此形成鲜明对比的是，NGD显式地纠正了这一几何效应，遵循理想的测地路径，并完全克服了GD所面临的不稳定性。我们通过实验证明，NGD不仅收敛速度显著更快，而且获得的解具有更优的泛化性能。这些结果确立了山脊的高度结构化几何最适合信息几何优化，为学习动力学与涌现表征几何之间的相互作用提供了新的视角。
英文摘要
High-capacity associative memories based on Kernel Logistic Regression (KLR) exhibit a "Ridge of Optimization" characterized by extreme stability and a highly skewed weight spectrum. However, the dynamical process by which learning converges to this critical regime has remained unclear. This paper provides a geometric analysis of the learning trajectories on the statistical manifold of a KLR-trained Hopfield network. By comparing the paths of Gradient Descent (GD) and Natural Gradient Descent (NGD), we elucidate the mechanisms governing the optimization process. Our analysis reveals that learning on the Ridge proceeds in two distinct phases. We show that the extreme curvature of the Ridge causes standard GD to follow a highly oscillatory, non-geodesic path. In stark contrast, NGD explicitly corrects for this geometry, following the ideal geodesic path and completely overcoming the instabilities faced by GD. We demonstrate experimentally that NGD not only converges significantly faster but also achieves a solution with superior generalization performance. These results establish that the highly structured geometry of the Ridge is optimally suited for information-geometric optimization, providing a new perspective on the interplay between learning dynamics and emergent representation geometry.
发表机构
- Aichi Institute of Technology（爱知工业大学）
机构由 AI 辅助整理，请以论文原文为准。
