arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.00535cs.LGcs.AIcs.CL
收藏

JoMA:通过MLP与注意力的联合动力学揭示多层Transformer的内在机制

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention

University of Washington(华盛顿大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Carnegie Mellon University(卡内基梅隆大学)

Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon Du

AI总结:

提出JoMA动力学框架,通过整合自注意力层简化多层Transformer训练分析,揭示注意力稀疏-密集变化规律,并在真实数据集和预训练模型上验证理论发现。

摘要:

我们提出了联合MLP/注意力(JoMA)动力学,这是一个新颖的数学框架,用于理解多层Transformer架构的训练过程。这是通过整合Transformer中的自注意力层实现的,仅产生MLP层的修正动力学。JoMA去除了先前分析中的不切实际假设(例如,缺乏残差连接),并预测在非线性激活存在的情况下,注意力首先变得稀疏(以学习显著标记),然后变得密集(以学习较不显著的标记),而在线性情况下,它与现有工作一致,表明注意力随时间变得稀疏。我们利用JoMA定性地解释了当输入标记由潜在层次生成模型生成时,多层Transformer中标记如何组合形成层次结构。在从真实世界数据集(Wikitext2/Wikitext103)训练的模型以及各种预训练模型(OPT、Pythia)上的实验验证了我们的理论发现。代码可在https://github.com/facebookresearch/luckmatters/tree/yuandong3找到。

英文摘要:

We propose Joint MLP/Attention (JoMA) dynamics, a novel mathematical framework to understand the training procedure of multilayer Transformer architectures. This is achieved by integrating out the self-attention layer in Transformers, producing a modified dynamics of MLP layers only. JoMA removes unrealistic assumptions in previous analysis (e.g., lack of residual connection) and predicts that the attention first becomes sparse (to learn salient tokens), then dense (to learn less salient tokens) in the presence of nonlinear activations, while in the linear case, it is consistent with existing works that show attention becomes sparse over time. We leverage JoMA to qualitatively explains how tokens are combined to form hierarchies in multilayer Transformers, when the input tokens are generated by a latent hierarchical generative model. Experiments on models trained from real-world dataset (Wikitext2/Wikitext103) and various pre-trained models (OPT, Pythia) verify our theoretical findings. Code can be found in https://github.com/facebookresearch/luckmatters/tree/yuandong3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑