两层网络中梯度下降重用批次的好处：打破信息指数与跳跃指数的诅咒
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
AI总结：
本文研究两层网络多索引目标函数学习中多次遍历梯度下降重用批次的训练动态，证明其能突破信息指数和跳跃指数限制，并刻画了有限时间内可高效学习的函数类。
摘要：
我们研究两层神经网络在学习多索引目标函数时的训练动态。我们重点关注多次遍历梯度下降（multi-pass gradient descent，GD），该方法会多次重用批次，并表明与单次遍历梯度下降相比，它显著改变了关于哪些函数可学习的结论。特别是，我们发现具有有限步长的多次遍历梯度下降能够克服由目标函数的信息指数（information exponent，Ben Arous et al., 2021）和跳跃指数（leap exponent，Abbe et al., 2023）所给出的梯度流和单次遍历梯度下降的局限性。我们证明，在重用批次后，即使对于不满足阶梯性质（staircase property，Abbe et al., 2021）的函数，网络也仅需两个时间步就能实现与目标子空间的重叠。我们刻画了在有限时间内可被高效学习的（广泛的）函数类。我们结果的证明基于动态平均场理论（Dynamical Mean-Field Theory，DMFT）的分析。我们进一步给出了权重低维投影动态过程的闭式描述，并提供了说明该理论的数值实验。
英文摘要：
We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about which functions are learnable compared to single-pass gradient descent. In particular, multi-pass GD with finite stepsize is found to overcome the limitations of gradient flow and single-pass GD given by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et al., 2023) of the target function. We show that upon re-using batches, the network achieves in just two time steps an overlap with the target subspace even for functions not satisfying the staircase property (Abbe et al., 2021). We characterize the (broad) class of functions efficiently learned in finite time. The proof of our results is based on the analysis of the Dynamical Mean-Field Theory (DMFT). We further provide a closed-form description of the dynamical process of the low-dimensional projections of the weights, and numerical experiments illustrating the theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
