Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Institute for Artificial Intelligence, Peking University, Beijing（人工智能研究院，北京大学，北京）；School of Integrated Circuits, Peking University, Beijing（集成电路学院，北京大学，北京）
Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla Attention, Sink Attention, and Gated Attention. We hope this study offers a new perspective on attention mechanisms and encourages further exploration of the inherent MoE structure within attention layers.
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China（复旦大学）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏滁研究中心）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（计算机科学系，科罗拉多大学博尔德分校）；Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.（浪潮 Genersoft 有限公司）；Inspur Group Co. Ltd.（浪潮集团有限公司）；Interactive Entertainment Group, Tencent Inc（腾讯公司互动娱乐集团）；Nanjing University of Information Science and Technology（南京信息工程大学）；Northeastern University（东北大学）
Continual Pre-training of MoEs: How robust is your router?
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究所）；Concordia University（康科迪亚大学）；University of Chicago（芝加哥大学）；Capital One（Capital One公司）
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Open University（上海开放大学）；ZhuQingTing Data Technology (Zhejiang) Co., Ltd（朱清庭数据科技（浙江）有限公司）
MoTE: Mixture of Task-specific Experts for Pre-Trained ModelBased Class-incremental Learning
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（信息与通信工程学院，北京邮电大学）；Engineering Research Center for Information Network, Ministry of Education（信息网络工程研究中心，教育部）
Dynamic Mixture of Latent Memories for Self-Evolving Agents
The Chinese University of Hong Kong（香港中文大学）；University of Oxford（牛津大学）；Nanyang Technological University（南洋理工大学）；University of Edinburgh（爱丁堡大学）；Shanghai Jiao Tong University（上海交通大学）
Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通大数据实验室，北京交通大学，教育部）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（模式识别中心，微信AI，腾讯公司，中国）
Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（中国科学技术大学认知智能国家重点实验室）；Meituan, Beijing（美团（北京））；AI for Science Institute, Beijing（北京人工智能科学研究院）
Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）
