Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that deep neural networks (DNNs) fit the target function from low to high frequency during the training, which provides insight into the training and generalization behavior of DNNs in complex tasks. In this paper, through analysis of an infinite-width two-layer NN in the neural tangent kernel (NTK) regime, we derive the exact differential equation, namely Linear Frequency-Principle (LFP) model, governing the evolution of NN output function in the frequency domain during the training. Our exact computation applies for general activation functions with no assumption on size and distribution of training data. This LFP model unravels that higher frequencies evolve polynomially or exponentially slower than lower frequencies depending on the smoothness/regularity of the activation function. We further bridge the gap between training dynamics and generalization by proving that LFP model implicitly minimizes a Frequency-Principle norm (FP-norm) of the learned function, by which higher frequencies are more severely penalized depending on the inverse of their evolution rate. Finally, we derive an \textit{a priori} generalization error bound controlled by the FP-norm of the target function, which provides a theoretical justification for the empirical results that DNNs often generalize well for low frequency functions.
Logic Synthesis Meets Machine Learning: Trading Exactness for Generalization
University of Tokyo(东京大学);Universidade Federal de Pelotas(佩洛塔斯联邦大学);National Taiwan University(台湾大学);University of Texas at Austin(得克萨斯大学奥斯汀分校);Universidade Federal do Rio Grande do Sul(南里奥格兰德联邦大学);Technische Universitaet Dresden(德累斯顿工业大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);IBM;Cornell University(康奈尔大学);Universidade Federal de Santa Catarina(圣卡塔琳娜联邦大学);University of Utah(犹他大学);UC Berkeley(加州大学伯克利分校);Google AI(谷歌人工智能)
Promoting Generalization for Exact Solvers via Adversarial Instance Augmentation
CAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China(GIPAS技术CAS重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
University of California, Berkeley(加州大学伯克利分校);Salesforce Research(Salesforce研究院)
最近的研究表明,经典的一致收敛界和零训练误差预测器(插值器)(如深度神经网络)的实际测试误差之间可能存在很大的差距。为了更好地理解这一差距,我们研究了非线性随机特征模型的一致收敛性,并对一致收敛如何依赖于样本大小和参数数量进行了精确的理论分析。我们推导并证明了该模型中三个量的解析表达式:1)范数球上的经典一致收敛,2)范数球上插值器上的一致收敛(最近由Zhou et al.(2020)提出),以及3)最小范数收敛的风险。我们证明,在经典一致收敛界为空(偏离$\infty$)的情况下,插值器的一致收敛仍然给出了插值解测试误差的非平凡界。我们还展示了一个不同的设置,经典的一致收敛界是非空的,但一致收敛插值可以提供一个改进的样本复杂度保证。我们的结果提供了第一个精确的比较之间的测试误差和一致收敛界的插值超越简单的线性模型。
Exact Mean Computation in Dynamic Time Warping Spaces
TU Berlin(柏林工业大学);Efficient Algorithms, TU Berlin(柏林工业大学高效算法);Algorithmics and Computational Complexity, TU Berlin(柏林工业大学算法与计算复杂性);Distributed Artificial Intelligence Laboratory, TU Berlin(柏林工业大学分布式人工智能实验室)