arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.13261cs.LG
收藏

面向Transformer中交叉头注意力的数据自适应马氏度量学习

Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers

Zhongnan University of Economics and Law(中南财经政法大学);ZUEL(中南财经政法大学);The Chinese University of Hong Kong(香港中文大学)

Zhenke Duan, Xin Li, Jiqun Pan, Dong Xiaofei, Hanwen Ning, Xinyuan Song

AI总结:

本文提出数据自适应马氏交叉头注意力(DAMCHA),通过学习完整交叉头度量矩阵与输入依赖的参数化方式改进Transformer的多头注意力,可作为即插即用替代方案,在基准任务上性能优于基线。

摘要:

本文提出了数据自适应马氏交叉头注意力(DAMCHA),这是一种简单有效的注意力机制,通过度量学习的视角对多头注意力(MHA)进行泛化。标准MHA隐式学习静态的、按头划分的欧氏相似度,对应统一交叉头度量矩阵的对角块,丢弃了所有头间的非对角交互。DAMCHA通过两项关键创新解决这一局限:其一,它学习完整的交叉头度量矩阵,诱导类马氏相似度,明确捕捉超出MHA受限块对角结构的头间交互;其二,它将度量矩阵参数化为依赖输入的函数,使注意力几何能动态适配数据的内在流形结构,而非在训练后保持固定。我们进一步引入了有原则的正则化策略,最显著的是逐层参数共享,以确保计算效率和稳定优化。DAMCHA可作为MHA及其变体在各类Transformer架构中的即插即用替代方案,在相当或更低的计算开销下提供更优的表达能力和学习性能。大量实验表明,基于DAMCHA的Transformer在一系列基准任务上始终优于竞争基线。代码可在该https URL获取。

英文摘要:

In this paper, we propose Data-Adaptive Mahalanobis Cross-Head Attention (DAMCHA), a simple and effective attention mechanism that generalizes Multi-Head Attention (MHA) through the lens of metric learning. Standard MHA implicitly learns a static, head-wise Euclidean similarity corresponding to the diagonal blocks of a unified cross-head metric matrix, discarding all off-diagonal interactions across heads. DAMCHA addresses this limitation via two key innovations. First, it learns the full cross-head metric matrix, inducing a Mahalanobis-like similarity that explicitly captures inter-head interactions beyond the restricted block-diagonal structure of MHA. Second, it parameterizes the metric matrix as an input-dependent function, enabling the attention geometry to dynamically adapt to the intrinsic manifold structure of the data rather than remaining fixed after training. We further introduce principled regularization strategies, most notably stack-wise parameter sharing, to ensure computational efficiency and stable optimization. DAMCHA serves as a drop-in replacement for MHA and its variants in various Transformer architectures, delivering superior expressive power and learning performance with comparable or reduced computational overhead. Extensive experiments demonstrate that DAMCHA-based Transformers consistently outperform competing baselines across a range of benchmark tasks. Code is available at https://github.com/erwinmsmith/DAMCHA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑