arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.13541cs.LGcs.AI
收藏

线性对数正态注意力与无偏集中性

Linear Log-Normal Attention with Unbiased Concentration

Huawei Technologies(华为技术有限公司)

Yury Nahshan, Joseph Kampeas, Emir Haleva

AI总结:

本文通过分析注意力矩阵的分布与集中性,提出线性对数正态注意力机制,在自然语言基准上优于其他线性化注意力方案,提升了Transformer处理长序列的可扩展性。

摘要:

Transformer模型在广泛的应用中取得了显著成果。然而,其可扩展性受到自注意力机制关于序列长度的二次时间和内存复杂度的制约。这一限制在处理长文档或高分辨率图像时构成了重大障碍。在本工作中,我们通过分析注意力矩阵的分布及其集中能力来研究自注意力机制。此外,我们提出了度量这些量的工具,并引入了一种新颖的自注意力机制——线性对数正态注意力(Linear Log-Normal Attention),旨在模拟原始自注意力的分布和集中行为。我们在流行的自然语言基准上的实验结果表明,我们提出的线性对数正态注意力优于其他线性化注意力替代方案,为增强Transformer模型的可扩展性提供了一条有前景的途径。

英文摘要:

Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑