arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11534cs.LGcs.AI
收藏

Krause同步变换器

Krause Synchronization Transformers

Shanghai Qi Zhi Institute(上海启智研究院);College of AI, Tsinghua University(清华大学人工智能学院);Shanghai Jiao Tong University(上海交通大学);California Institute of Technology(加州理工学院);University of Amsterdam(阿姆斯特丹大学)

Jingkun Liu, Yisong Yue, Max Welling, Yue Song

AI总结:

提出基于有界置信共识动力学的Krause注意力机制,通过局部化稀疏交互替代全局softmax归一化,缓解表示坍缩和注意力汇聚现象,实现线性复杂度并提升性能。

摘要:

Transformer中的自注意力依赖于全局归一化的softmax权重,导致所有token在每一层竞争影响力。当跨深度组合时,这种交互模式会诱导强同步动力学,倾向于收敛到主导模式,这种行为与表示坍缩和注意力汇聚现象相关。我们引入了Krause注意力,一种受有界置信共识动力学启发的原则性注意力机制。Krause注意力将基于相似性的全局聚合替换为基于距离的、局部化的、选择性稀疏的交互,促进结构化的局部同步而非全局混合。我们将这种行为与最近将Transformer动力学建模为相互作用粒子系统的理论联系起来,并展示有界置信交互如何自然地调节注意力集中并缓解注意力汇聚。将交互限制在局部邻域还将运行时复杂度从序列长度的二次方降低到线性。实验上,我们在多种设置中验证了Krause注意力,包括视觉(CIFAR/ImageNet上的ViT)、自回归图像生成(MNIST/CIFAR-10)、大语言模型(Llama/Qwen)以及从零开始训练的多种规模(100M/200M)的语言模型。在这些领域中,Krause注意力在提高计算效率的同时实现了持续的性能提升,突显了有界置信动力学作为注意力的一种可扩展且有效的归纳偏置。

英文摘要:

Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that favor convergence toward a dominant mode, a behavior associated with representation collapse and attention sink phenomena. We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics. Krause Attention replaces similarity-based global aggregation with distance-based, localized, and selectively sparse interactions, promoting structured local synchronization instead of global mixing. We relate this behavior to recent theory modeling Transformer dynamics as interacting particle systems, and show how bounded-confidence interactions naturally moderate attention concentration and alleviate attention sinks. Restricting interactions to local neighborhoods also reduces runtime complexity from quadratic to linear in sequence length. Empirically, we validate Krause Attention across diverse settings, including vision (ViT on CIFAR/ImageNet), autoregressive image generation (MNIST/CIFAR-10), large language models (Llama/Qwen), and language models trained from scratch at multiple scales (100M/200M). Across these domains, Krause Attention achieves consistent performance gains while improving computational efficiency, highlighting bounded-confidence dynamics as a scalable and effective inductive bias for attention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑