arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.19611cs.LGcs.AIcs.CL
收藏

显式多头注意力机制用于大语言模型中头间的交互

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models

Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学)

Runyu Peng, Yunhua Zhou, Demin Song, Kai Lv, Bo Wang, Qipeng Guo, Xipeng Qiu

AI总结:

本文提出显式多头注意力机制,通过显式建模头间交互提升注意力性能,实现更高效的参数利用和内存压缩,同时保持模型性能。

摘要:

在基于Transformer架构构建的大语言模型中,近期研究表明头间交互可以提升注意力性能。受此启发,我们提出了多头显式注意力(MEA),这是一种简单而有效的注意力变体,能够显式建模跨头交互。MEA由两个关键组件组成:一个在头级别应用可学习线性组合的Head-level Linear Composition(HLC)模块,该模块对跨头的键向量和值向量分别应用可学习的线性组合,从而实现丰富的头间通信;以及一个头级别组归一化层,该层对重新组合后的头进行统计特性对齐。MEA在预训练中表现出强大的鲁棒性,这允许使用更大的学习率,从而导致更快的收敛,最终实现更低的验证损失和在多种任务上的性能提升。此外,我们通过减少注意力头的数量,并利用HLC来使用低秩“虚拟头”重建它们,探索了MEA的参数效率。这使我们能够实现一种实用的键值缓存压缩策略,该策略在知识密集型和科学推理任务中将KV缓存内存使用量减少50%,同时性能损失可以忽略不计,并且在奥林匹克级别的数学基准测试中仅导致3.59%的准确率下降。

英文摘要:

In large language models built upon the Transformer architecture, recent studies have shown that inter-head interaction can enhance attention performance. Motivated by this, we propose Multi-head Explicit Attention (MEA), a simple yet effective attention variant that explicitly models cross-head interaction. MEA consists of two key components: a Head-level Linear Composition (HLC) module that separately applies learnable linear combinations to the key and value vectors across heads, thereby enabling rich inter-head communication; and a head-level Group Normalization layer that aligns the statistical properties of the recombined heads. MEA shows strong robustness in pretraining, which allows the use of larger learning rates that lead to faster convergence, ultimately resulting in lower validation loss and improved performance across a range of tasks. Furthermore, we explore the parameter efficiency of MEA by reducing the number of attention heads and leveraging HLC to reconstruct them using low-rank "virtual heads". This enables a practical key-value cache compression strategy that reduces KV-cache memory usage by 50% with negligible performance loss on knowledge-intensive and scientific reasoning tasks, and only a 3.59% accuracy drop for Olympiad-level mathematical benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑