频率原理:傅里叶分析揭示深度神经网络
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
AI总结:
该研究通过傅里叶分析揭示深度神经网络训练中从低频到高频拟合的普遍频率原理,解释了其良好泛化及在奇偶函数等数据上的失败原因。
摘要:
我们从傅里叶分析的角度研究深度神经网络(DNNs)的训练过程。我们证明了一个非常普遍的频率原理(F-Principle)——DNNs 通常从低频到高频拟合目标函数——这一原理在高维基准数据集(如 MNIST/CIFAR10)和深度神经网络(如 VGG16)上均成立。DNNs 的这一频率原理与大多数传统迭代数值方法(如 Jacobi 方法)的行为相反,后者在各种科学计算问题中对高频表现出更快的收敛速度。通过一个简单的理论,我们说明这一频率原理源于常用激活函数的正则性。频率原理暗示了一种隐式偏差:DNNs 倾向于用低频函数拟合训练数据。这一理解解释了 DNNs 在大多数真实数据集上具有良好的泛化能力,而在奇偶函数或随机数据集上泛化能力较差的原因。
英文摘要:
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。