arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.02010cs.LG
收藏

Transformer中多头注意力的记忆容量

Memorization Capacity of Multi-Head Attention in Transformers

University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(矢量人工智能研究院)

Sadegh Mahdavi, Renjie Liao, Christos Thrampoulidis

AI总结:

本文研究Transformer多头注意力的记忆容量,基于输入线性无关假设证明含H头、维度d且n<d的注意力层可记忆Ω(Hn)个示例,并通过合成实验验证。

摘要:

Transformer已成为语言和视觉任务的首选架构,但其理论性质,尤其是记忆容量,仍不明确。本文研究多头注意力机制的记忆能力,考察其能够记忆多少个示例序列,并将其表示为注意力头数量和序列长度的函数。受视觉Transformer实验结果的启发,本文提出了关于输入数据线性无关的新假设,这不同于常用的一般位置假设。在这些假设下,本文证明:一个具有H个头、维度d、上下文大小n<d且包含Θ(Hd²)个参数的注意力层,可以记忆Ω(Hn)个示例。借助softmax算子的饱和性质,本文分析阐明了不同注意力头如何处理各种示例序列。本文通过在合成数据上的实验验证了这些发现。

英文摘要:

Transformers have become the go-to architecture for language and vision tasks, yet their theoretical properties, especially memorization capacity, remain elusive. This paper investigates the memorization abilities of multi-head attention mechanisms, examining how many example sequences they can memorize, as a function of the number of heads and sequence length. Motivated by experimental findings on vision transformers, we introduce novel assumptions about the linear independence of input data, distinct from the commonly used general-position assumption. Under these assumptions, we demonstrate that an attention layer with $H$ heads, dimension $d$, and context size $n < d$, featuring $Θ(Hd^2)$ parameters, can memorize $Ω(Hn)$ examples. Our analysis sheds light on how different attention heads handle various example sequences, aided by the softmax operator's saturation property. We validate our findings through experiments on synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑