The Pennsylvania State University(宾夕法尼亚州立大学);University of Connecticut(康涅狄格大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Los Angeles(加州大学洛杉矶分校)
Long-context inference in large language models is bottlenecked by Key--Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High-Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi-Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from a sharding bottleneck during distributed decoding via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced to redundantly load the complete KV cache for every token, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Extensive experiments show that MLRA achieves state-of-the-art perplexity and downstream task performance, while also delivering a 2.8$\times$ decoding speedup over MLA. Code is available at https://github.com/SongtaoLiu0823/MLRA. Pretrained weights, along with the training and evaluation data, are available at https://huggingface.co/Soughing/MLRA.
An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction
Zhejiang University(浙江大学);Tianjin Jinhang Technical Physics Institute(天津津航技术物理研究所);Complex System Control and Intelligent Agent Cooperative Laboratory(复杂系统控制与智能体协同实验室)
Quantum Graph Attention Network: A Novel Quantum Multi-Head Attention Mechanism for Graph Learning
An Ning School of Electrical Engineering(电气工程学院);Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);National Center for High-performance Computing (NCHC)(高性能计算中心)
MoH: Multi-Head Attention as Mixture-of-Head Attention
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China(电子与计算机工程系,深圳研究生院,北京大学,深圳,中国);Pengcheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国);School of AI for Science, Shenzhen Graduate School, Peking University, Shenzhen, China(科学人工智能学院,深圳研究生院,北京大学,深圳,中国);National University of Singapore, Singapore(新加坡国立大学,新加坡)
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Institute for Artificial Intelligence, Peking University, Beijing(人工智能研究院,北京大学,北京);School of Integrated Circuits, Peking University, Beijing(集成电路学院,北京大学,北京)
The University of Hong Kong(香港大学);Sun Yat-sen University(中山大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Transformers在各种重要任务上推进了自然语言处理(NLP)领域。Transformer体系结构的基石是多头注意力(MHA)机制,它对序列元素之间的成对交互进行建模。尽管取得了巨大的成功,但目前的框架忽略了不同头部之间的相互作用,导致许多头部在实践中是多余的,这极大地浪费了模型的容量。为了提高参数效率,我们重新制定的MHA作为一个潜在的变量模型从概率的角度来看。我们提出了级联头碰撞注意力(CODA),显式模型之间的相互作用,通过层次变分分布的注意头。我们进行了大量的实验,并证明CODA优于Transformer基线,通过在语言建模中的\texttt{Wikitext-103}上的$0.6$困惑,以及在机器翻译中的\texttt{WMT14 EN-DE}上的$0.6$ BLEU,由于它在参数效率上的改进。\footnote{Our implementation is publicly available at \url{https://github.com/LZhengisme/CODA}.}
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Krembil Research Institute, University Health Network(肯贝尔研究所,大学健康网络)
Multimodal Multi-Head Convolutional Attention with Various Kernel Sizes for Medical Image Super-Resolution
University of Bucharest(布加勒斯特大学);“Carol Davila” University of Medicine and Pharmacy(“卡罗尔·达维拉”医药大学);Colţea Hospital(科尔察医院);University Politehnica of Bucharest(布加勒斯特理工大学);MBZ University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Linköping University(林雪平大学)
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(越南国家大学胡志明市分校科学大学);Vietnam National University, Ho Chi Minh City(越南国家大学胡志明市分校);Trivita AI
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Department of Computer Science, Birla Institute of Technology(比拉理工学院计算机科学系);School of Computer Engineering, Kalinga Institute of Industrial Technology(凯林加工业技术学院计算机工程系);Birla AI Labs, Office of Ananya Birla, Aditya Birla Group(比拉人工智能实验室,阿尼娅·比拉办公室,阿迪塔亚比拉集团)