Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell, \mathcal{F},\mathsf{P})$ ERM can have slow $(1/\sqrt{n})$ or fast $(1/n)$ rates of convergence of the excess risk as a function of the sample size $n$. There exist several results that give sufficient conditions for fast rates in terms of joint properties of $\ell$, $\mathcal{F}$, and $\mathsf{P}$, such as the margin condition and the Bernstein condition. In the non-statistical prediction with expert advice setting, there is an analogous slow and fast rate phenomenon, and it is entirely characterized in terms of the mixability of the loss $\ell$ (there being no role there for $\mathcal{F}$ or $\mathsf{P}$). The notion of stochastic mixability builds a bridge between these two models of learning, reducing to classical mixability in a special case. The present paper presents a direct proof of fast rates for ERM in terms of stochastic mixability of $(\ell,\mathcal{F}, \mathsf{P})$, and in so doing provides new insight into the fast-rates phenomenon. The proof exploits an old result of Kemperman on the solution to the general moment problem. We also show a partial converse that suggests a characterization of fast rates for ERM in terms of stochastic mixability is possible.
Mixability made efficient: Fast online multiclass logistic regression
INRIA(法国国家信息与自动化研究所);PSL Research University(巴黎文理研究大学);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立理工学院);LJK(约瑟夫·傅立叶实验室)
可混性已被证明是一个强大的工具,以获得最佳的遗憾算法。然而,所得到的方法通常遭受高计算复杂性,这降低了它们的实用性。例如,在多类逻辑回归的情况下,聚合预测器(Foster et al.(2018))实现了$O(\log(Bn))$的遗憾,而在线牛顿步骤实现了$O(e^B\log(n))$,获得了$B$的双指数增益(比较函数的范数的界限)。然而,这种高统计性能是以过高的计算复杂性$O(n^{37})$为代价的。
Stochastic Cubic Regularization for Fast Nonconvex Optimization
University of California, Berkeley(加州大学伯克利分校)
本文提出了一个经典算法的随机变体-正则化牛顿法[Nesterov and Polyak 2006]。该算法有效地逃脱鞍点,并找到近似的局部极小值一般光滑,非凸函数在只有$\mathcal{\tilde{O}}(ε^{-3.5})$随机梯度和随机海森向量积的评价。后者可以像随机梯度一样有效地计算。这改进了随机梯度下降的$\mathcal{\tilde{O}}(ε^{-4})$速率。我们的速度匹配最著名的结果,找到当地的最小值,而不需要任何微妙的加速或方差减少技术。
Stochastic Gradient Descent: Going As Fast As Possible But Not Faster
UCSF(加州大学旧金山分校);INRIA-CNRS-UPSud-UPSay(法国国家信息与自动化研究所-法国国家科学研究中心-巴黎南大学-巴黎萨克雷大学);TAU, U. Paris-Sud(计算机科学研究实验室,巴黎南大学);CNRS-UPSud-INRIA-UPSay(法国国家科学研究中心-巴黎南大学-法国国家信息与自动化研究所-巴黎萨克雷大学)
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
University of British Columbia(英属哥伦比亚大学);Mila, Université de Montréal(蒙特利尔大学米拉研究所);Element AI(元素人工智能公司);CCAI Affiliate Chair (Amii)(加拿大先进人工智能研究所CCAI附属主席)
Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization
University of Pittsburgh(匹兹堡大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);MIIT Key Laboratory of Pattern Analysis & Machine Intelligence(工业和信息化部模式分析与机器智能重点实验室);JD Finance America Corporation(京东金融美国公司)
FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding
Harvard University(哈佛大学);Franklin and Marshall College(富兰克林与马歇尔学院)
块浮点(BFP)可以通过经由跨一组值的共享指数提供宽动态范围来有效地支持深度神经网络(DNN)训练的量化。在本文中,我们提出了一种用于DNN的快速第一,准确第二训练(FAST)系统,其中权重,激活和梯度以BFP表示。FAST支持使用可变精度BFP输入操作数的矩阵乘法,从而在整个训练过程中实现DNN精度的增量增加。通过提高训练迭代和DNN层的BFP精度,FAST可以大大缩短训练时间,同时减少整体硬件资源使用。我们的FAST乘法器-累加器(fMAC)支持多个BFP精度下的点积计算。我们在具有不同数据集的多个DNN上验证了我们的FAST系统,在基于\textbf{mixed-precision or block}浮点数系统的先前工作中,在单芯片平台上的训练中展示了2-6个$\times$加速,同时在验证准确性方面实现了类似的性能。
Stochastic Auto-conditioned Fast Gradient Methods with Optimal Rates
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, 225 North Ave, Atlanta, GA 30332, USA(希尔-米利-斯图尔特工业与系统工程学院,佐治亚理工学院,225北大道,亚特兰大,GA 30332,USA)