Attention并非你所需要的全部：纯Attention的秩随深度以双重指数速度丢失
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
AI总结：
本工作提出一种分解方法证明纯self-attention网络对“token uniformity”具有强烈inductive bias，若无skip connections或MLPs，其输出会以双重指数速度收敛至rank-1矩阵，并通过实验验证了该收敛现象。
摘要：
基于attention的架构在机器学习中已变得无处不在，然而我们对其有效性的原因的理解仍然有限。本工作提出了一种理解self-attention网络的新方法：我们证明其输出可分解为更小项的总和，每项都涉及跨层的一系列attention head的操作。利用该分解，我们证明self-attention对“token uniformity”具有强烈的inductive bias。具体而言，如果没有skip connections或multi-layer perceptrons (MLPs)，输出会以双重指数速度收敛至rank-1矩阵。另一方面，skip connections和MLPs能阻止输出退化。我们的实验在标准transformer架构的不同变体上验证了所识别的收敛现象。
英文摘要：
Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
