Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
University of Alberta(阿尔伯塔大学);Mila, McGill University(米拉研究所,麦吉尔大学);University of British Columbia(不列颠哥伦比亚大学);Cornell University(康奈尔大学);Mila, Université de Montréal(米拉研究所,蒙特利尔大学)
Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, Simon Lacoste-Julien
Adaptive gradient methods are typically used for training over-parameterized models. To better understand their behaviour, we study a simplistic setting -- smooth, convex losses with models over-parameterized enough to interpolate the data. In this setting, we prove that AMSGrad with constant step-size and momentum converges to the minimizer at a faster $O(1/T)$ rate. When interpolation is only approximately satisfied, constant step-size AMSGrad converges to a neighbourhood of the solution at the same rate, while AdaGrad is robust to the violation of interpolation. However, even for simple convex problems satisfying interpolation, the empirical performance of both methods heavily depends on the step-size and requires tuning, questioning their adaptivity. We alleviate this problem by automatically determining the step-size using stochastic line-search or Polyak step-sizes. With these techniques, we prove that both AdaGrad and AMSGrad retain their convergence guarantees, without needing to know problem-dependent constants. Empirically, we demonstrate that these techniques improve the convergence and generalization of adaptive gradient methods across tasks, from binary classification with kernel mappings to multi-class classification with deep networks.
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
最近一系列关于深度学习的研究集中在极端过度参数化的设置上,并表明当网络宽度大于训练样本大小$n$和目标误差$ε^{-1}$的倒数的高次多项式时,通过(随机)梯度下降学习的深度神经网络具有良好的优化和泛化保证。最近,研究表明,在对训练数据的某些裕度假设下,多对数宽度条件足以使双层ReLU网络收敛和泛化(Ji and Telgarsky,2019)。然而,深度神经网络是否可以通过这种温和的过度参数化来学习仍然是一个悬而未决的问题。在这项工作中,我们肯定地回答了这个问题,并为通过(随机)梯度下降训练的深度ReLU网络建立了更清晰的学习保证。具体而言,在一定的假设下,在以前的工作中,我们的优化和推广保证持有$n$和$ε^{-1}$的网络宽度多对数。我们的研究结果将过度参数化深度神经网络的研究推向了更实用的设置。
A Convergence Theory for Deep Learning via Over-Parameterization
Microsoft Research AI(微软研究院AI);Stanford University(斯坦福大学);Princeton University(普林斯顿大学);UT-Austin(德克萨斯大学奥斯汀分校);University of Washington(华盛顿大学);Harvard University(哈佛大学)
Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
Amazon/A9(亚马逊/A9)
梯度消失和梯度爆炸是训练深度神经网络的两个主要障碍,特别是在捕获递归神经网络~(RNN)中的长程依赖关系方面。在本文中,我们提出了RNN的转移矩阵的有效参数化,使我们能够稳定其训练中出现的梯度。具体来说,我们通过奇异值分解(SVD)来参数化转移矩阵,这允许我们显式地跟踪和控制其奇异值。我们通过使用数值线性代数中常见的工具来实现效率,即Householder反射器用于表示SVD中出现的正交矩阵。通过显式控制奇异值,我们提出的Spectral-RNN方法使我们能够轻松解决梯度爆炸问题,并且我们观察到它在很大程度上凭经验解决了梯度消失问题。我们注意到SVD参数化可以用于任何矩形权重矩阵,因此它可以很容易地扩展到任何深度神经网络,例如多层感知器。从理论上讲,我们证明了我们的参数化不会失去任何表达能力,并展示了它如何控制RNN的分类任务的泛化。%,并展示它如何潜在地使优化过程更容易。我们广泛的实验结果也表明,所提出的框架收敛速度更快,具有良好的泛化能力,特别是在捕捉长范围的依赖关系,如合成添加和复制任务,以及MNIST和Penn Tree Bank数据集。
Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection
University of British Columbia(不列颠哥伦比亚大学);University of California, Davis(加州大学戴维斯分校);Dato
最近在随机坐标下降算法的理论和应用方面已经有了重要的工作,开始于Nesterov [SIAM J. Optim.,22(2),2012],其示出了随机坐标选择规则实现与高斯-索斯韦尔选择规则相同的收敛速度。这个结果表明,我们不应该使用高斯-索斯韦尔规则,因为它通常比随机选择昂贵得多。然而,这些算法的经验行为与这一理论结果相矛盾:在选择规则的计算成本相当的应用中,高斯-索斯韦尔选择规则往往比随机坐标选择表现得更好。我们给出了一个简单的分析高斯Southwell规则显示-除了在极端情况下-其收敛速度是快于选择随机坐标。此外,在这项工作中,我们(i)表明,精确的坐标优化提高了某些稀疏问题的收敛速度,(ii)提出了一个高斯-Southwell-Lipschitz规则,给出了一个更快的收敛速度给定知识的Lipschitz常数的偏导数,(iii)分析近似高斯-Southwell规则的效果,(iv)分析近似梯度的高斯-Southwell规则的变体。
Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
Department of Computing Science, University of Alberta(谷歌DeepMind);School of Data Science, The Chinese University of Hong Kong, Shenzhen;School of Computational Science;Google DeepMind;CIFAR AI Chair, Amii;The work was done while the author was at Google.
Special-Unitary Parameterization for Trainable Variational Quantum Circuits
Department of Electrical and Electronic Engineering, Imperial College London, London, UK(电气与电子工程系,帝国理工学院伦敦校区,伦敦,英国);Centre for Quantum Engineering, Science and Technology (QuEST), Imperial College London, London, UK(量子工程、科学与技术中心(QuEST),帝国理工学院伦敦校区,伦敦,英国);Brookhaven National Laboratory, New York, USA(布鲁克海文国家实验室,纽约,美国);National Taiwan University, Taipei, Taiwan(台湾大学,台北,台湾)