arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16153cs.LGstat.ML
收藏

论注意力层中秩的益处

On the Benefits of Rank in Attention Layers

Courant Institute of Mathematical Sciences, New York University(纽约大学柯朗数学科学研究所);Center for Data Science, New York University(纽约大学数据科学中心);Flatiron Institute(弗拉蒂隆研究所)

Noah Amsel, Gilad Yehudai, Joan Bruna

AI总结:

本文揭示了注意力机制中矩阵秩与头数间的显著权衡,证明单头全秩可表示的函数需指数级多头低秩才能近似,短上下文可借深度补偿,并经实验验证。

摘要:

基于注意力的机制在机器学习中被广泛使用,最显著的是在transformers中。然而,诸如注意力矩阵的秩和头数等超参数在该架构的所有实现中几乎都以相同的方式进行缩放,而没有理论依据。在这项工作中,我们展示了注意力机制的秩和头数之间存在显著的权衡。具体而言,我们提出了一个简单且自然的目标函数,对于任意上下文长度,都可以使用单个全秩注意力头来表示,但除非头数在嵌入维度上呈指数级增长,否则无法通过低秩注意力来近似,即使对于短上下文长度也是如此。此外,我们证明,对于短上下文长度,增加深度允许低秩注意力来近似该目标。对于长上下文,我们推测全秩注意力是必要的。最后,我们使用现成的transformers进行了实验,验证了我们的理论发现。

英文摘要:

Attention-based mechanisms are widely used in machine learning, most prominently in transformers. However, hyperparameters such as the rank of the attention matrices and the number of heads are scaled nearly the same way in all realizations of this architecture, without theoretical justification. In this work we show that there are dramatic trade-offs between the rank and number of heads of the attention mechanism. Specifically, we present a simple and natural target function that can be represented using a single full-rank attention head for any context length, but that cannot be approximated by low-rank attention unless the number of heads is exponential in the embedding dimension, even for short context lengths. Moreover, we prove that, for short context lengths, adding depth allows the target to be approximated by low-rank attention. For long contexts, we conjecture that full-rank attention is necessary. Finally, we present experiments with off-the-shelf transformers that validate our theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑