在基于Transformer架构构建的大语言模型中,近期研究表明头间交互可以提升注意力性能。受此启发,我们提出了多头显式注意力(MEA),这是一种简单而有效的注意力变体,能够显式建模跨头交互。MEA由两个关键组件组成:一个在头级别应用可学习线性组合的Head-level Linear Composition(HLC)模块,该模块对跨头的键向量和值向量分别应用可学习的线性组合,从而实现丰富的头间通信;以及一个头级别组归一化层,该层对重新组合后的头进行统计特性对齐。MEA在预训练中表现出强大的鲁棒性,这允许使用更大的学习率,从而导致更快的收敛,最终实现更低的验证损失和在多种任务上的性能提升。此外,我们通过减少注意力头的数量,并利用HLC来使用低秩“虚拟头”重建它们,探索了MEA的参数效率。这使我们能够实现一种实用的键值缓存压缩策略,该策略在知识密集型和科学推理任务中将KV缓存内存使用量减少50%,同时性能损失可以忽略不计,并且在奥林匹克级别的数学基准测试中仅导致3.59%的准确率下降。
英文摘要:
In large language models built upon the Transformer architecture, recent studies have shown that inter-head interaction can enhance attention performance. Motivated by this, we propose Multi-head Explicit Attention (MEA), a simple yet effective attention variant that explicitly models cross-head interaction. MEA consists of two key components: a Head-level Linear Composition (HLC) module that separately applies learnable linear combinations to the key and value vectors across heads, thereby enabling rich inter-head communication; and a head-level Group Normalization layer that aligns the statistical properties of the recombined heads. MEA shows strong robustness in pretraining, which allows the use of larger learning rates that lead to faster convergence, ultimately resulting in lower validation loss and improved performance across a range of tasks. Furthermore, we explore the parameter efficiency of MEA by reducing the number of attention heads and leveraging HLC to reconstruct them using low-rank "virtual heads". This enables a practical key-value cache compression strategy that reduces KV-cache memory usage by 50% with negligible performance loss on knowledge-intensive and scientific reasoning tasks, and only a 3.59% accuracy drop for Olympiad-level mathematical benchmarks.
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
MoH: Multi-Head Attention as Mixture-of-Head Attention
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China(电子与计算机工程系,深圳研究生院,北京大学,深圳,中国);Pengcheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国);School of AI for Science, Shenzhen Graduate School, Peking University, Shenzhen, China(科学人工智能学院,深圳研究生院,北京大学,深圳,中国);National University of Singapore, Singapore(新加坡国立大学,新加坡)
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Institute for Artificial Intelligence, Peking University, Beijing(人工智能研究院,北京大学,北京);School of Integrated Circuits, Peking University, Beijing(集成电路学院,北京大学,北京)
The University of Hong Kong(香港大学);Sun Yat-sen University(中山大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Transformers在各种重要任务上推进了自然语言处理(NLP)领域。Transformer体系结构的基石是多头注意力(MHA)机制,它对序列元素之间的成对交互进行建模。尽管取得了巨大的成功,但目前的框架忽略了不同头部之间的相互作用,导致许多头部在实践中是多余的,这极大地浪费了模型的容量。为了提高参数效率,我们重新制定的MHA作为一个潜在的变量模型从概率的角度来看。我们提出了级联头碰撞注意力(CODA),显式模型之间的相互作用,通过层次变分分布的注意头。我们进行了大量的实验,并证明CODA优于Transformer基线,通过在语言建模中的\texttt{Wikitext-103}上的$0.6$困惑,以及在机器翻译中的\texttt{WMT14 EN-DE}上的$0.6$ BLEU,由于它在参数效率上的改进。\footnote{Our implementation is publicly available at \url{https://github.com/LZhengisme/CODA}.}
Safety Alignment Should Be Made More Than Just A Few Attention Heads
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Krembil Research Institute, University Health Network(肯贝尔研究所,大学健康网络)
Identifying Semantic Induction Heads to Understand In-Context Learning
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Baidu Inc.(百度公司)
An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction
Zhejiang University(浙江大学);Tianjin Jinhang Technical Physics Institute(天津津航技术物理研究所);Complex System Control and Intelligent Agent Cooperative Laboratory(复杂系统控制与智能体协同实验室)
Quantum Graph Attention Network: A Novel Quantum Multi-Head Attention Mechanism for Graph Learning
An Ning School of Electrical Engineering(电气工程学院);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);National Center for High-performance Computing (NCHC)(高性能计算中心)