arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.07628cs.LGcs.DS
收藏

次二次时间内训练多层过参数化神经网络

Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time

Adobe Research(奥多比研究院);Massachusetts Institute of Technology(麻省理工学院);Simons Institute for the Theory of Computing(西蒙斯计算理论研究所)

Zhao Song, Lichen Zhang, Ruizhe Zhang

AI总结:

本文提出一种框架,在初始化阶段仅用 $m^2$ 成本,将多层过参数化神经网络每次迭代训练成本降至真正的次二次 $m^{2-\Omega(1)}$,并为加速大语言模型微调提供了启示。

摘要:

我们考虑训练一个多层过参数化神经网络以最小化由损失函数导出的经验风险的问题。在典型的过参数化设置中,网络宽度 $m$ 远大于数据维度 $d$ 和训练样本数 $n$($m=\mathrm{poly}(n,d)$),这导致每层产生一个极其庞大的权重矩阵 $W\in \mathbb{R}^{m\times m}$。朴素地,在前向和反向计算中,必须花费 $O(m^2)$ 的时间来读取权重矩阵并评估神经网络函数。在本研究中,我们展示了如何降低每次迭代的训练成本。具体而言,我们提出了一个框架,仅在初始化阶段使用 $m^2$ 的成本,并在 $m$ 的意义上实现了每次迭代真正的次二次成本,即每次迭代 $m^{2-\Omega(1)}$。我们的结果具有超越标准过参数化理论的意义,因为它可以被视为在预训练大模型之上设计高效数据结构,以进一步加速微调过程,这是部署大语言模型(LLM)的核心步骤。

英文摘要:

We consider the problem of training the last layer of a multi-layer over-parametrized neural network to minimize the empirical risk induced by a loss function. In the typical setting of over-parametrization, the network width $m$ is much larger than the data dimension $d$ and the number of training samples $n$ ($m=\mathrm{poly}(n,d)$), which induces a prohibitively large weight matrix $W\in\mathbb{R}^{m\times m}$ per layer above the first. Naively, when the layer inputs are dense, one has to pay $O(m^2)$ time to read the weight matrix and evaluate the neural network function in both forward and backward computation. To reduce the training cost per iteration, we propose a framework that uses $O(m^2nL)$ time in the initialization phase and achieves a truly subquadratic amortized cost per iteration in terms of $m$, i.e., $m^{2-Ω(1)}$ per iteration. The speedup comes from a sparse activation, a shifted ReLU whose threshold at each layer is proportional to the norm of the layer input. With this threshold every layer is positively homogeneous, so our convergence proof needs a width only polynomial in the depth $L$, in $n$ and in the inverse smallest kernel eigenvalues. The inputs of the trained layer are fixed and sparse. In this setting, simpler algorithms without a data structure reach the same cost per iteration, and maintaining the products of the last layer with its inputs lowers it to $mn^{ω-1+o(1)}$, where $ω$ is the matrix multiplication exponent. Our result can be viewed as an idealized model of maintaining a small amount of state on top of a frozen pre-trained model to speed up fine-tuning, a core procedure to deploy large language models (LLMs). Our guarantees assume that the frozen layers are random, and for a pre-trained model only the cost per iteration of maintaining these products carries over.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑