Shanghai Qi Zhi Institute(上海启智研究院);College of AI, Tsinghua University(清华大学人工智能学院);Shanghai Jiao Tong University(上海交通大学);California Institute of Technology(加州理工学院);University of Amsterdam(阿姆斯特丹大学)
Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that favor convergence toward a dominant mode, a behavior associated with representation collapse and attention sink phenomena. We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics. Krause Attention replaces similarity-based global aggregation with distance-based, localized, and selectively sparse interactions, promoting structured local synchronization instead of global mixing. We relate this behavior to recent theory modeling Transformer dynamics as interacting particle systems, and show how bounded-confidence interactions naturally moderate attention concentration and alleviate attention sinks. Restricting interactions to local neighborhoods also reduces runtime complexity from quadratic to linear in sequence length. Empirically, we validate Krause Attention across diverse settings, including vision (ViT on CIFAR/ImageNet), autoregressive image generation (MNIST/CIFAR-10), large language models (Llama/Qwen), and language models trained from scratch at multiple scales (100M/200M). Across these domains, Krause Attention achieves consistent performance gains while improving computational efficiency, highlighting bounded-confidence dynamics as a scalable and effective inductive bias for attention.
Random Quadratic Form on a Sphere: Synchronization by Common Noise
Korteweg-de Vries Institute for Mathematics, University of Amsterdam(阿姆斯特丹大学Korteweg-de Vries数学研究所);Department of Mathematics and Computer Science, FU Berlin(柏林自由大学数学与计算机科学系)
Kuramoto Attention: Synchronizing Self-Attention on the Torus
Department of Informatics, Luddy School of Informatics, Computing, and Engineering, Cognitive Science Program, Indiana University Bloomington(印第安纳大学伯明顿分校信息学系,卢迪信息学、计算与工程学院,认知科学项目)
Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series
Department of Biomedical Engineering, PolyU(PolyU 生物医学工程系);School of Nursing, PolyU(PolyU 护理学院);Yau Mathematical Sciences Center, Tsinghua University(清华大学尤太数学科学中心)
Interpreting the Synchronization Gap: The Hidden Mechanism Inside Diffusion Transformers
Leinweber Institute for Theoretical Physics;Department of Physics, University of California, Berkeley, CA, 94720-7300, USA;Theoretical Physics Group, Lawrence Berkeley National Laboratory, Berkeley, CA 94720-8162, USA;Department of Radiation Oncology, University of California, San Francisco;Computational Precision Health, University of California, San Francisco;Department of Physics, University of California, Berkeley, CA 94720, USA
Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks
Pillar of Information Systems Technology and Design, Singapore University of Technology and Design(信息系统技术与设计学院,新加坡南洋理工大学);School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast(电子、电气与计算机科学学院,女王大学贝尔法斯特分校);Department of Electrical and Computer Engineering, Sungkyunkwan University(电气与计算机工程系,成均馆大学);KU 6G Research Center, Department of Computer and Information Engineering, Khalifa University(6G研究中心,计算机与信息工程系,哈利法大学);CentraleSupelec, University of Paris-Saclay(中央超导研究所,巴黎萨克雷大学);Department of Electrical and Computer Engineering, Princeton University(电气与计算机工程系,普林斯顿大学)
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学先进交叉学科学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(中国科学院人工智能安全国家重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);School of Mathematics, Southeast University, Nanjing, China(东南大学数学学院)
From TLinFormer to TConstFormer: The Leap to Constant-Time Transformer Attention: Achieving O(1) Computation and O(1) KV Cache during Autoregressive Inference
Synchronization on circles and spheres with nonlinear interactions
The Wharton School, University of Pennsylvania, USA(沃森学校,宾夕法尼亚大学);Institute of Mathematics, EPFL, Lausanne, Switzerland(数学研究所,EPFL,拉沃斯纳,瑞士);CERMICS, ENPC, Institut Polytechnique de Paris, CNRS, France(CERMICS,ENPC,巴黎理工学院,CNRS,法国)
The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
Coordination Among Neural Modules Through a Shared Global Workspace
Mila(米拉人工智能研究所);University of Montreal(蒙特利尔大学);Google Deepmind(谷歌DeepMind);Max Planck Institute(马克斯·普朗克研究所);Google Research(谷歌研究院);Microsoft Research(微软研究院);Indian Institute of Technology, Delhi(印度理工学院德里分校)