arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.08153cs.LG
收藏

线性频率原理动力学的精确计算及其推广

On the exact computation of linear frequency principle dynamics and its generalization

Shanghai Jiao Tong University(上海交通大学)

Tao Luo, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang

摘要:

最近的研究表明,深度神经网络(DNN)在训练过程中从低频到高频拟合目标函数的频率原理(F-Principle)现象非常有趣,这为深入了解DNN在复杂任务中的训练和泛化行为提供了新的视角。本文通过对神经切线核(NTK)结构下的无限宽两层神经网络的分析,推导出了神经网络训练过程中输出函数在频域演化的精确微分方程,即线性频率原理(LFP)模型。我们的精确计算适用于一般的激活函数没有假设的大小和分布的训练数据。该LFP模型揭示了取决于激活函数的平滑度/规律性,较高频率比较低频率多项式或指数地慢地演变。我们通过证明LFP模型隐式地最小化学习函数的频率原则范数(FP范数),进一步弥合了训练动态和泛化之间的差距,根据其进化速率的倒数,更高的频率受到更严重的惩罚。最后,我们得到了一个由目标函数的FP-范数控制的\textit{a priori}泛化误差界,这为DNN通常对低频函数泛化良好的经验结果提供了理论依据。

英文摘要:

Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the training, which provides insight into the training and generalization behavior of DNNs in complex tasks. In this paper, through analysis of an infinite-width two-layer NN in the neural tangent kernel (NTK) regime, we derive the exact differential equation, namely Linear Frequency-Principle (LFP) model, governing the evolution of NN output function in the frequency domain during the training. Our exact computation applies for general activation functions with no assumption on size and distribution of training data. This LFP model unravels that higher frequencies evolve polynomially or exponentially slower than lower frequencies depending on the smoothness/regularity of the activation function. We further bridge the gap between training dynamics and generalization by proving that LFP model implicitly minimizes a Frequency-Principle norm (FP-norm) of the learned function, by which higher frequencies are more severely penalized depending on the inverse of their evolution rate. Finally, we derive an \textit{a priori} generalization error bound controlled by the FP-norm of the target function, which provides a theoretical justification for the empirical results that DNNs often generalize well for low frequency functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑