多头注意力（MHA）是现代大型语言模型（LLMs）的核心计算基本单元。然而，MHA存在根本性的线性扩展限制：H个注意力头会产生恰好H个独立的注意力矩阵，在注意力计算过程中头之间没有交流。这在多步推理中变得有问题，因为正确的答案依赖于从上下文的多个部分聚合证据，并在中间推断的链式中组成潜在的token到token的关系。为了解决这个问题，我们提出了交错头部注意力（IHA），它通过为每个头构造P个伪头（通常P=H）来实现跨头混合，其中每个伪查询/键/值是所有H个原始查询、键和值的学得线性组合。伪查询和伪键头之间的交互在每个头中诱导出多达P²种注意力模式，参数开销为O(H²P)。我们提供了理论证明，显示在合成多项式任务（IHA使用Θ(√k n²)参数 vs. MHA的Θ(kn²)）和合成顺序敏感的CPM-3任务（IHA使用⌈√N_max⌉个头 vs. MHA的N_max）中，效率得到了改进。在真实世界基准测试中，IHA在RULER上的多键检索上提高了10-20%（4k-16k），并在OpenThoughts上经过推理微调后，GSM8K提高了5.8%，MATH-500提高了2.8%（多数投票）。
英文摘要：
Multi-Head Attention (MHA) is the core computational primitive underlying modern Large Language Models (LLMs). However, MHA suffers from a fundamental linear scaling limitation: $H$ attention heads produce exactly $H$ independent attention matrices, with no communication between heads during attention computation. This becomes problematic for multi-step reasoning, where correct answers depend on aggregating evidence from multiple parts of the context and composing latent token-to-token relations over a chain of intermediate inferences. To address this, we propose Interleaved Head Attention (IHA), which enables cross-head mixing by constructing $P$ pseudo-heads per head (typically $P=H$), where each pseudo query/key/value is a learned linear combination of all $H$ original queries, keys and values respectively. Interactions between pseudo-query and pseudo-key heads induce up to $P^2$ attention patterns per head with modest parameter overhead $\mathcal{O}(H^2P)$. We provide theory showing improved efficiency in terms of number of parameters on the synthetic Polynomial task (IHA uses $Θ(\sqrt{k}n^2)$ parameters vs. $Θ(kn^2)$ for MHA) and on the synthetic order-sensitive CPM-3 task (IHA uses $\lceil\sqrt{N_{\max}}\rceil$ heads vs. $N_{\max}$ for MHA). On real-world benchmarks, IHA improves Multi-Key retrieval on RULER by 10-20% (4k-16k) and, after fine-tuning for reasoning on OpenThoughts, improves GSM8K by 5.8% and MATH-500 by 2.8% (Majority Vote) over full attention.
An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction
Zhejiang University（浙江大学）；Tianjin Jinhang Technical Physics Institute（天津津航技术物理研究所）；Complex System Control and Intelligent Agent Cooperative Laboratory（复杂系统控制与智能体协同实验室）
Quantum Graph Attention Network: A Novel Quantum Multi-Head Attention Mechanism for Graph Learning
An Ning School of Electrical Engineering（电气工程学院）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；National Center for High-performance Computing (NCHC)（高性能计算中心）
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Department of Computer Science, Birla Institute of Technology（比拉理工学院计算机科学系）；School of Computer Engineering, Kalinga Institute of Industrial Technology（凯林加工业技术学院计算机工程系）；Birla AI Labs, Office of Ananya Birla, Aditya Birla Group（比拉人工智能实验室，阿尼娅·比拉办公室，阿迪塔亚比拉集团）
MoH: Multi-Head Attention as Mixture-of-Head Attention
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China（电子与计算机工程系，深圳研究生院，北京大学，深圳，中国）；Pengcheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）；School of AI for Science, Shenzhen Graduate School, Peking University, Shenzhen, China（科学人工智能学院，深圳研究生院，北京大学，深圳，中国）；National University of Singapore, Singapore（新加坡国立大学，新加坡）
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Institute for Artificial Intelligence, Peking University, Beijing（人工智能研究院，北京大学，北京）；School of Integrated Circuits, Peking University, Beijing（集成电路学院，北京大学，北京）
The University of Hong Kong（香港大学）；Sun Yat-sen University（中山大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Transformers在各种重要任务上推进了自然语言处理（NLP）领域。Transformer体系结构的基石是多头注意力（MHA）机制，它对序列元素之间的成对交互进行建模。尽管取得了巨大的成功，但目前的框架忽略了不同头部之间的相互作用，导致许多头部在实践中是多余的，这极大地浪费了模型的容量。为了提高参数效率，我们重新制定的MHA作为一个潜在的变量模型从概率的角度来看。我们提出了级联头碰撞注意力（CODA），显式模型之间的相互作用，通过层次变分分布的注意头。我们进行了大量的实验，并证明CODA优于Transformer基线，通过在语言建模中的\texttt{Wikitext-103}上的$0.6$困惑，以及在机器翻译中的\texttt{WMT14 EN-DE}上的$0.6$ BLEU，由于它在参数效率上的改进。\footnote{Our implementation is publicly available at \url{https://github.com/LZhengisme/CODA}.}
HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beihang University（北京航空航天大学）；National Cheng-Kung University（国立成功大学）
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
