arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01763cs.LGcs.AIcs.CC
收藏

混合线性-全注意力的可证明表达性层级

A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention

Taylor Lab, Huawei(华为泰勒实验室);Key Laboratory of Interdisciplinary Research of Computation and Economics(计算与经济学交叉研究关键实验室);Shanghai University of Finance and Economics(上海财经大学)

Xiaowei Ye, Xiaoyu He, Chao Liao, Chen Wu, Pinyan Lu

AI总结:

本文通过理论分析,揭示了混合注意力与全注意力在表达能力上的差异,证明了混合注意力在解决多步推理任务时的局限性。

摘要:

Transformer 作为大多数现代大语言模型的基础。为了缓解标准全注意力的二次复杂度,各种高效的注意力机制,如线性注意力和混合注意力,已被开发。一个根本的差距仍然存在:它们与全注意力的表达能力缺乏严格的理论描述。在本工作中,我们理论上描述了这些注意力机制之间的性能差异。我们的理论适用于所有可以表示为递归的线性注意力变体,包括 Mamba、DeltaNet 等。具体来说,我们建立了表达性层级:对于序列函数组合-多步推理任务,必须在模型的前向传递中发生,一个(L+1)层全注意力网络是足够的,而任何混合网络交替使用 L-1 层全注意力和大量(2^{3L^2})线性注意力层都无法解决该任务。这一结果展示了两种注意力类型在表达能力上的明显分离。我们的工作提供了混合注意力与标准全注意力之间的首次可证明分离,为理解不同注意力机制的基本能力和限制提供了理论视角。

英文摘要:

Transformers serve as the foundation of most modern large language models. To mitigate the quadratic complexity of standard full attention, various efficient attention mechanisms, such as linear and hybrid attention, have been developed. A fundamental gap remains: their expressive power relative to full attention lacks a rigorous theoretical characterization. In this work, we theoretically characterize the performance differences among these attention mechanisms. Our theory applies to all linear attention variants that can be formulated as a recurrence, including Mamba, DeltaNet, etc. Specifically, we establish an expressiveness hierarchy: for the sequential function composition-a multi-step reasoning task that must occur within a model's forward pass, an ($L+1$)-layer full attention network is sufficient, whereas any hybrid network interleaving $L-1$ layers of full attention with a substantially larger number ($2^{3L^2}$) of linear attention layers cannot solve it. This result demonstrates a clear separation in expressive power between the two types of attention. Our work provides the first provable separation between hybrid attention and standard full attention, offering a theoretical perspective for understanding the fundamental capabilities and limitations of different attention mechanisms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑