发表机构
Institute of Physics and Astronomy, University of Potsdam（波茨坦大学物理与天文学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文以正则化强度为可调参数，通过解析研究深度线性神经网络玩具模型，揭示了正则化诱导的特征检测相变级联与损失景观几何的关联，为深度学习科学理论提供了研究平台。
AI 中文摘要
深度学习的科学理论，涵盖学习动态与已学习模型的统计特性，正迅速受到关注。该领域发展的基石之一是可解析求解的玩具模型，其可对学习动态进行完全可处理的分析。本文中，我们以正则化强度作为可调外部参数（类似统计物理学中的外场），对这类玩具模型展开解析研究。过往研究中，（i）已通过解析预测到学习转变的 onset（起始点），（ii）已通过现象学/数值方式证实，调节正则化强度可引发一系列相变级联，且这类相变的数量与由模型复杂度决定的损失景观几何结构相关。我们构建了支撑过往数值观测的严谨框架，研究揭示了这类相变级联、可学习特征与 underlying（ underlying 译为“ underlying 保持”）几何结构间的精确关联。我们给出了这些相变的解析预测，以及与已学习特征相关的可处理序参量。在最小模型层面，我们将这种可浓缩为有效描述的宏观视角，与由 Hessian 谱表征的损失景观几何结构所体现的微观视角相连接。因此，本文提出的模型为探索和深化基于统计物理学概念的深度学习科学理论进展提供了平台。
英文摘要
A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention. One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics. Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics. In previous studies, (i) an onset of learning transition was predicted analytically and (ii) it was phenomenologically/numerically established that tuning the regularization strength can result in a cascade of phase transitions. The number of those transitions was linked to the geometry of the loss landscape determined by the model complexity. Setting up a rigorous framework underpinning the previous numerical observations, our investigation reveals a precise connection between those cascades of phase transitions, learnable features and the underlying geometry. We provide analytic predictions of these phase transitions as well as tractable order parameters related to learned features. At the level of the minimal model, we connect this macroscopic perspective (that can be condensed into an effective description) to the microscopic perspective in terms of the geometry of the loss landscape characterized by the Hessian spectrum. Thus, the presented model provides a platform to explore and sharpen advances made in the scientific theory of deep learning rooted in statistical physics concepts.
Comments39 pages, 9 figures
