arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.05554cs.LGcs.AIcs.DMmath.CA
收藏

长上下文Transformer中的关键注意力缩放

Critical attention scaling in long-context transformers

Department of Mathematics, Massachusetts Institute of Technology(数学系,麻省理工学院);Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(电气工程与计算机科学系,麻省理工学院)

Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet

AI总结:

该研究针对长上下文Transformer的注意力秩崩溃问题,通过分析简化模型确定关键缩放因子$\beta_n \backsim \text{log} n$,为YaRN和Qwen的注意力缩放提供理论依据,阐明对数缩放可维持长上下文下的稀疏内容自适应注意力。

摘要:

随着大语言模型扩展至更长的上下文,注意力层出现了一个根本性问题:注意力分数随上下文长度$n$增大而趋于均匀,导致 token 过度聚类,这一现象被称为秩崩溃。虽然注意力缩放通过用多对数因子$\beta_n$重新缩放注意力分数能有效解决这一缺陷,但该方法仍缺乏理论依据。我们分析了一个简化且易处理的模型,该模型放大了注意力缩放的效果。在该模型中,注意力呈现由缩放因子$\beta_n$控制的相变:缩放不足会将所有 token 崩溃至单一方向,而缩放过度则会使注意力退化为恒等变换,从而消除 token 间有意义的交互。我们的主要结果确定了关键缩放因子$\beta_n \backsim \boldsymbol{\text{log}} \boldsymbol{n}$,并为 YaRN 和 Qwen 中的注意力缩放提供了严格的理论依据,阐明了对数缩放为何能在长上下文长度下维持稀疏、内容自适应的注意力机制。

英文摘要:

As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse. While $\textit{attention scaling}$ effectively addresses this deficiency by rescaling attention scores with a polylogarithmic factor $β_n$, theoretical justification for this approach remains lacking. We analyze a simplified yet tractable model that magnifies the effect of attention scaling. In this model, attention exhibits a phase transition governed by the scaling factor $β_n$: insufficient scaling collapses all tokens to a single direction, while excessive scaling reduces attention to identity, thereby eliminating meaningful interactions between tokens. Our main result identifies the critical scaling $β_n \asymp \log n$ and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑