arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04032cs.LGcs.AIcs.CLcs.PF
收藏

Transformer 需要三个投影吗?QKV 变体的系统研究

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Ali Kayyam;Anusha Madan Gopal;M Anthony Lewis

Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis

AI总结:

本文系统研究了注意力机制中查询、键、值投影共享的变体,发现 Q-K=V 共享在语言建模中仅以 3.1% 的困惑度损失实现 50% 的 KV 缓存减少,且与头共享结合可达到 96.9% 的缓存减少,从而支持设备端推理。

摘要:

Transformer 已成为各种 AI 任务的标准解决方案,其中查询、键和值(QKV)注意力公式起着核心作用。然而,这三个投影的各自贡献以及省略某些投影的影响仍知之甚少。我们系统评估了三种投影共享约束:a) Q-K=V(共享键-值),b) Q=K-V(共享查询-键),c) Q=K=V(单投影)。后两种变体产生对称注意力图;为了解决这个问题,我们还通过二维位置编码探索了非对称注意力。通过涵盖合成任务、视觉(MNIST、CIFAR、TinyImageNet、异常检测)和语言建模(在 10B 令牌上训练的 300M 和 1.2B 参数模型)的实验,我们发现我们的 Transformer 性能与 QKV Transformer 相当,有时甚至更好。在语言建模中,Q-K=V 投影共享实现了 50% 的 KV 缓存减少,仅导致 3.1% 的困惑度下降。关键的是,投影共享与头共享(GQA/MQA)互补:将 Q-K=V 与 GQA-4 结合可实现 87.5% 的缓存减少,而 Q-K=V + MQA 则达到 96.9%,从而实现了实用的设备端推理。我们表明,Q-K=V 保持了质量,因为键和值可以占据相似的表示空间,并且注意力在低秩机制下运行,而 Q=K-V 则破坏了注意力的方向性。我们的结果系统地将投影共享描述为注意力中权重绑定的一种未被充分探索的实例,具有直接、可量化的推理内存优势,尤其对边缘部署有价值。代码公开于 https://github.com/anushamadan02/Do-Transformers-Need-3-Projections。

英文摘要:

Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at https://github.com/Brainchip-Inc/Do-Transformers-Need-3-Projections

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑