arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01334math.OC
收藏

非线性扰动阈值:深度学习中网络宽度缩放和景观分岔

The Non-Linearity Perturbation Threshold: Width Scaling and Landscape Bifurcations in Deep Learning

oeaw.ac.at(奥地利科学院)

Michael Alexander

AI总结:

研究神经网络优化景观在引入非线性激活函数时的变形,通过光滑同调分析,揭示局部拓扑变化由经典codimension-1正常形式控制,并证明双线性过参数化在临界点产生Hessian核,激活函数同伦软化该阈值,得出分岔点lambda*与网络宽度成正比。

摘要:

我们研究了当非线性激活函数通过光滑同伦引入时,神经网络优化景观的变形。首先在抽象的局部环境中,即一个光滑的一参数函数族以及一个失去非退化性的临界分支,通过Lyapunov-Schmidt约简,我们显示局部过渡由经典的codimension-1正常形式(transcritical或pitchfork)控制,并且相关的拓扑变化由Morse理论的handle附件所支配。然后我们超越抽象框架,验证这些假设对于具体的两层架构。我们证明双线性过参数化在线性端点产生(m-1)d维的Hessian核,Tikhonov正则化将其提升到一个floor alpha > 0;激活函数同伦软化这个floor,产生显式的分岔点lambda*大约等于alpha/|lambda_1'(0)|。我们推导了特征值软化率作为激活导数和数据矩的函数,并证明近pitchfork正常形式(|g_aa/g_aaa|远小于1)是tanh-like激活函数sigma''(0)=0的结构结果。分岔点的规模与网络宽度成正比,将框架连接到NTK领域:在大m时,景观重组被推过lambda=1,线性化的图景占主导地位。

英文摘要:

We study how the optimization landscape of a neural network deforms as a non-linear activation is introduced through a smooth homotopy. Working first in an abstract local setting - a smooth one-parameter family of objective functions together with a critical branch that loses non-degeneracy through a simple Hessian kernel - we show via Lyapunov-Schmidt reduction that the local transition is controlled by the classical codimension-one normal forms (transcritical or pitchfork) and that the associated topology change is governed by Morse-theoretic handle attachment. We then move beyond the abstract framework and verify these assumptions for a concrete two-layer architecture. We prove that bilinear overparameterization creates an (m-1)d-dimensional Hessian kernel at the linear endpoint, which Tikhonov regularization lifts to a floor alpha > 0; the activation homotopy softens this floor, yielding an explicit bifurcation point lambda* approximately equal to alpha/|lambda_1'(0)|. We derive the eigenvalue-softening rate as a functional of activation derivatives and data moments, and prove that the near-pitchfork normal form (|g_aa/g_aaa| much less than 1) is a structural consequence of sigma''(0)=0 for tanh-like activations. The bifurcation point scales as lambda* proportional to alpha m with network width, connecting the framework to the NTK regime: at large m the landscape reorganization is pushed past lambda=1 and the linearized picture prevails. The foundational algebraic theorems have been formally verified in the Lean 4 theorem prover, and theoretical predictions computed for widths m in {3, 5, 10, 20, 50, 100} exhibit quantitative agreement with the abstract framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑