In this paper, we propose Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a simple and effective attention mechanism that generalizes Multi-Head Attention (MHA) through the lens of metric learning. Standard MHA implicitly learns a static, head-wise Euclidean similarity corresponding to the diagonal blocks of a unified cross-head metric matrix, discarding all off-diagonal interactions across heads. DAMCHA addresses this limitation via two key innovations. First, it learns the full cross-head metric matrix, inducing a Mahalanobis-like similarity that explicitly captures inter-head interactions beyond the restricted block-diagonal structure of MHA. Second, it parameterizes the metric matrix as an input-dependent function, enabling the attention geometry to dynamically adapt to the intrinsic manifold structure of the data rather than remaining fixed after training. We further introduce principled regularization strategies, most notably stack-wise parameter sharing, to ensure computational efficiency and stable optimization. DAMCHA serves as a drop-in replacement for MHA and its variants in various Transformer architectures, delivering superior expressive power and learning performance with comparable or reduced computational overhead. Extensive experiments demonstrate that DAMCHA-based Transformers consistently outperform competing baselines across a range of benchmark tasks. Code is available at https://github.com/erwinmsmith/DAMCHA.
Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning
School of Mathematics and Statistics, Chongqing Technology and Business University(重庆工商大学数学与统计学院);Chongqing Key Laboratory of Statistical Intelligent Computing and Monitoring, Chongqing Technology and Business University(重庆工商大学统计智能计算与监测重庆市重点实验室);School of Food Science, Chongqing Technology and Business University(重庆工商大学食品科学与工程学院);Faculty of Electrical Engineering and Information Technology, TU Dortmund University(多特蒙德工业大学电气工程与信息技术学院)
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏滁研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Krembil Research Institute, University Health Network(肯贝尔研究所,大学健康网络)
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
Massachusetts Institute of Technology(麻省理工学院);École Polytechnique Fédérale de Lausanne(瑞士联邦理工学院);University of California, Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学)
Improving Transformers with Probabilistic Attention Keys
FPT Software AI Center(FPT软件人工智能中心);University of California, Los Angeles(加州大学洛杉矶分校);VinUniversity(文大);Deezer Research(Deezer研究院);Rice University(莱斯大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
多头注意力是最先进的Transformers背后的驱动力,它在各种自然语言处理(NLP)和计算机视觉任务中实现了卓越的性能。已经观察到,对于许多应用,这些注意力头学习冗余嵌入,并且其中大多数可以在不降低模型性能的情况下被移除。受此启发,我们提出了Transformer与混合高斯密钥(transformer-MGK),一种新的Transformer架构,取代冗余的头在每个头的混合密钥的变压器。这些键的混合遵循高斯混合模型,并允许每个注意力头有效地集中在输入序列的不同部分。与传统的Transformer相比,Transformer-MGK加速了训练和推理,参数更少,需要计算的FLOP更少,同时在任务中实现了相当或更好的准确性。transformer-MGK也可以很容易地扩展到使用线性注意力。我们经验证明了Transformer-MGK在一系列实际应用中的优势,包括语言建模和涉及很长序列的任务。在Wikitext-103和Long Range Arena基准测试中,具有4个头的Transformer-MGK获得了与具有8个头的基线Transformers相当或更好的性能。
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Center for Data Science(数据科学中心);Peking University(北京大学);Institute for Functional Intelligent Materials(功能智能材料研究所);National University of Singapore(新加坡国立大学);Department of Mathematics(数学系);Microsoft Research Asia(微软亚洲研究院)
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
The University of Tokyo(东京大学);RIKEN AIP(理化学研究所AIP);Flatiron Institute(Flatiron研究所);CNRS(法国国家科学研究中心);Université Côte d’Azur(蔚蓝海岸大学);New York University(纽约大学)
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国);School of Software, Soongsil University, Seoul, South Korea(软件学院,顺天大学,首尔,韩国)