arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.13233cs.LGcs.AIcs.CL
收藏

多层Transformer的梯度可以在近线性时间内近似计算

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time

The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学);The Simons Institute for the Theory of Computing at the University of California, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所);University of Pennsylvania(宾夕法尼亚大学)

Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, Yufa Zhou

AI总结:

本文针对多层Transformer模型自注意力机制计算梯度存在的二次时间复杂度瓶颈,提出一种新颖的快速近似方法,在近线性时间内计算梯度并保持多项式级微小误差,适用于包含残差连接等子模块的一般损失函数,有望促进长上下文语言模型的高效训练与部署。

摘要:

流行transformer架构中自注意力机制的计算复杂度对训练和推理构成了重大挑战,并成为长输入的瓶颈。是否有可能显著降低多层transformer模型中计算梯度的二次时间复杂度?本文证明了一种新颖的快速近似方法,能够在近线性时间 $n^{1+o(1)}$ 内计算梯度,其中 $n$ 为输入序列长度,同时在整个模型中保持多项式级微小的近似误差 $1 / \mathrm{poly}(n)$。我们的理论适用于一般的损失函数,且当多层transformer模型包含诸多实用子模块时同样成立,例如残差连接、因果掩码和多头注意力。通过提升梯度计算的效率,我们希望这项工作能基于理论结果促进长上下文语言模型更高效的训练与部署。

英文摘要:

The computational complexity of the self-attention mechanism in popular transformer architectures poses significant challenges for training and inference, and becomes the bottleneck for long inputs. Is it possible to significantly reduce the quadratic time complexity of computing the gradients in multi-layer transformer models? This paper proves that a novel fast approximation method can calculate the gradients in almost linear time $n^{1+o(1)}$ where $n$ is the input sequence length, while it maintains a polynomially small approximation error $1 / \mathrm{poly}(n)$ across the entire model. Our theory holds for general loss functions and when the multi-layer transformer model contains many practical sub-modules, such as residual connection, casual mask, and multi-head attention. By improving the efficiency of gradient computation, we hope that this work will facilitate more effective training and deployment of long-context language models based on our theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑