ReLU闸门重尾现象的实验研究
An Empirical Study of the Occurrence of Heavy-Tails in Training a ReLU Gate
摘要：
最近关于随机深度学习算法的一个特殊进展方向是揭示这些算法的平稳分布的一个相当神秘的重尾性质，即使数据分布不是这样。此外，厚尾指数是已知的，以显示有趣的依赖于网络的输入尺寸，小批量的大小和算法的步长。在这篇短文中，我们对S.G.D.的这个指数进行了实验研究。同时训练$\relu$门（在可实现的和二进制分类设置中），并且对于S.G.D. Karmakar和Mukherjee（2022）证明了ReLU可实现数据。从我们的实验中，我们推测，这两个算法有类似的重尾行为的任何数据，后者可以被证明是收敛的。其次，我们证明了在这个模型的情况下，晚时间迭代的重尾指数与线性假设类已经证明的或以前已经证明的大型网络具有显着不同的属性。
英文摘要：
A particular direction of recent advance about stochastic deep-learning algorithms has been about uncovering a rather mysterious heavy-tailed nature of the stationary distribution of these algorithms, even when the data distribution is not so. Moreover, the heavy-tail index is known to show interesting dependence on the input dimension of the net, the mini-batch size and the step size of the algorithm. In this short note, we undertake an experimental study of this index for S.G.D. while training a $\relu$ gate (in the realizable and in the binary classification setup) and for a variant of S.G.D. that was proven in Karmakar and Mukherjee (2022) for ReLU realizable data. From our experiments we conjecture that these two algorithms have similar heavy-tail behaviour on any data where the latter can be proven to converge. Secondly, we demonstrate that the heavy-tail index of the late time iterates in this model scenario has strikingly different properties than either what has been proven for linear hypothesis classes or what has been previously demonstrated for large nets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
