Predicting the outputs of finite deep neural networks trained with noisy gradients
Racah Institute of Physics, Hebrew University（希伯来大学拉卡物理研究所）；Edmond and Lily Safra Center for Brain Sciences, Hebrew University（希伯来大学埃德蒙与莉莉·萨夫拉脑科学中心）；Center for Brain Science, Harvard University（哈佛大学脑科学中心）；Agilent Research Labs（安捷伦研究实验室）
Gadi Naveh, Oded Ben-David, Haim Sompolinsky, Zohar Ringel
A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs). A DNN trained with gradient flow was shown to map to a GP governed by the Neural Tangent Kernel (NTK), whereas earlier works showed that a DNN with an i.i.d. prior over its weights maps to the so-called Neural Network Gaussian Process (NNGP). Here we consider a DNN training protocol, involving noise, weight decay and finite width, whose outcome corresponds to a certain non-Gaussian stochastic process. An analytical framework is then introduced to analyze this non-Gaussian process, whose deviation from a GP is controlled by the finite width. Our contribution is three-fold: (i) In the infinite width limit, we establish a correspondence between DNNs trained with noisy gradients and the NNGP, not the NTK. (ii) We provide a general analytical form for the finite width correction (FWC) for DNNs with arbitrary activation functions and depth and use it to predict the outputs of empirical finite networks with high accuracy. Analyzing the FWC behavior as a function of $n$, the training set size, we find that it is negligible for both the very small $n$ regime, and, surprisingly, for the large $n$ regime (where the GP error scales as $O(1/n)$). (iii) We flesh out algebraically how these FWCs can improve the performance of finite convolutional neural networks (CNNs) relative to their GP counterparts on image classification tasks.
Learning with Noisy Labels via Sparse Regularization
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）
带噪声标签的学习对于训练准确的深度神经网络来说是一项重要且具有挑战性的任务。一些常用的损失函数，如交叉熵（CE），遭受严重的过拟合噪声标签。满足对称条件的鲁棒损失函数被定制以解决这个问题，然而，这遇到了欠拟合效应。在本文中，我们从理论上证明了\textbf{any loss can be made robust to noisy labels}通过限制网络输出的一组排列在一个固定的向量。当固定向量是one-hot时，我们只需要将输出约束为one-hot，但这几乎在所有地方都产生零梯度，从而使基于梯度的优化变得困难。在这项工作中，我们引入了稀疏正则化策略来近似one-hot约束，该策略由网络输出锐化操作和$\ell_p$ -norm（$p\le 1$）正则化组成，该锐化操作使网络的输出分布变得尖锐，该正则化使网络的输出变得稀疏。这种简单的方法保证了任意损失函数的鲁棒性，同时不妨碍拟合能力。实验结果表明，该方法在噪声标签和类别不平衡的情况下，能够显著提高常用损失函数的性能，优于现有的方法.该代码可在https://github.com/hitcszx/lnl_sr上获得。
Mediated Uncoupled Learning: Learning Functions without Direct Input-output Correspondences
CNRS（法国国家科学研究中心）；Université Paris-Dauphine, PSL Research University（巴黎多芬大学，巴黎文理研究大学）；RIKEN AIP（理化学研究所先进智能项目中心）；Kyoto University（京都大学）；The University of Tokyo（东京大学）
Generalization Error Bounds for Noisy, Iterative Algorithms
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）
在统计学习理论中，泛化误差用于量化监督机器学习算法可能过拟合训练数据的程度。最近的工作[Xu and Raginsky（2017）]基于算法输入$S$和算法输出$W$之间的互信息$I(S;W)$，当损失函数为亚高斯时，建立了经验风险最小化的泛化误差的界。我们利用这些结果推导出泛化误差界的一大类迭代算法，其特征在于有界的，嘈杂的更新与马尔可夫结构。我们的边界是非常普遍的，适用于许多设置的兴趣，包括随机梯度朗之万动力学（SGLD）和随机梯度哈密顿蒙特卡罗（SGHMC）算法的变种。此外，我们的误差界适用于在迭代路径上计算的任何输出函数，包括算法的最后一个子过程或迭代子集的平均值，并且还允许在算法的连续更新中对数据进行非均匀采样。
