In order to elucidate the plateau phenomena caused by vanishing gradient, we herein analyse stability of stochastic gradient descent near degenerated subspaces in a multi-layer perceptron. In stochastic gradient descent for Fukumizu-Amari model, which is the minimal multi-layer perceptron showing non-trivial plateau phenomena, we show that (1) attracting regions exist in multiply degenerated subspaces, (2) a strong plateau phenomenon emerges as a noise-induced synchronisation, which is not observed in deterministic gradient descent, (3) an optimal fluctuation exists to minimise the escape time from the degenerated subspace. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks.
Parameter-Free Online Convex Optimization with Sub-Exponential Noise
Boston University(波士顿大学)
考虑了带次指数噪声的无约束在线凸优化问题,这是一个比标准的在线凸优化更一般的问题。在这种设置中,学习器接收被次指数噪声破坏的损失函数的次梯度,并努力实现最佳后悔保证,而不知道竞争者范数,即,in a parameter-free参数of way.最近,Cutkosky和Boahen(COLT 2017)证明,给定无界次梯度,由于指数惩罚,不可能保证次线性后悔。本文表明,它是可能的去周围的下限,允许所观察到的次梯度是无界通过随机噪声。然而,在无约束OCO中存在无界噪声是具有挑战性的,现有的算法不提供接近最优的遗憾界限或不能保证。因此,我们设计了一个新的参数自由OCO算法的Banach空间,我们称之为BANCO,通过减少对噪声硬币的投注。我们表明,BANCO实现了最佳的遗憾率在我们的问题。最后,我们展示了我们的结果的应用,以获得一个无参数的局部私有随机次梯度下降算法,并连接到法律的迭代算法。
Google AI Princeton(谷歌AI普林斯顿);Princeton University(普林斯顿大学);University of Washington(华盛顿大学);Allen School of Computer Science and Engineering(阿伦计算机科学与工程学院)
University of Technology Sydney(悉尼科技大学);National University of Singapore(新加坡国立大学);City University of Hong Kong(香港城市大学);Global Business College of Australia(澳大利亚全球商学院)
Online Parameter Identification of Generalized Non-cooperative Game
Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学);Tongji University(同济大学);Shanghai Research Institute for Intelligent Autonomous Systems(上海自主智能无人系统科学中心)
Online Convex Optimization with Switching Cost with Only One Single Gradient Evaluation
Indian Institute of Technology Bombay (IITB)(印度理工学院班加罗尔分校);Indian Institute of Technology Delhi (IITD)(印度理工学院德里分校);Tata Institute of Fundamental Research (TIFR)(塔塔基础研究机构)