arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.01251cs.LGcs.AIcs.ITmath.ITmath.STstat.MLstat.TH
收藏

频域深度神经网络的训练行为

Training behavior of deep neural network in frequency domain

Shanghai Jiao Tong University(上海交通大学);Institute for Advanced Study(普林斯顿高等研究院)

Zhi-Qin John Xu, Yaoyu Zhang, Yanyang Xiao

摘要:

为什么能够过拟合的深度神经网络(DNN)在实践中通常可以很好地泛化是一个谜[#zhang2016understanding]。为了找到一个潜在的机制,我们专注于研究潜在的DNN的训练过程中的内隐偏见。在这项工作中,对于真实和合成数据集,我们根据经验发现,具有常见设置的DNN首先快速捕获占主导地位的低频分量,然后相对缓慢地捕获高频分量。我们称这种现象为频率原理(F原理)。在我们的实验中,可以在各种结构、激活函数和训练算法的DNN上观察到F原则。我们还说明了F原则如何帮助理解早期停止的效果以及DNN的泛化。这个F原则可能提供了对DNN优化和泛化的一般原则的见解。

英文摘要:

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑