分组查询专家：GQA自注意力上的混合专家
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
AI总结：
提出GQE，在分组查询注意力上应用混合专家层，每token仅激活一半查询头，在250M参数规模下匹配全激活基线性能。
摘要：
自注意力是Transformer性能的核心，并且在长上下文长度下通常是Transformer中最昂贵的部分，因为其成对token交互随序列长度呈二次方缩放。标准的密集注意力还对每个token应用相同的注意力头集，无论token难度或信息内容如何。这种均匀激活会浪费计算，尤其是当序列变长且注意力成本迅速增加时。我们提出了分组查询专家（GQE），一种在分组查询注意力（GQA）之上的混合专家层。在每个GQA组内，一个路由器为每个token选择k个查询头专家，而所有键值（KV）头保持密集且不变。因此，GQE保留了GQA的KV缓存优势，并仅减少活跃的查询头计算。在250M参数规模下，固定30B token预算，GQE在下游准确率上与全激活GQA基线匹配，同时每token仅激活一半的查询头。
英文摘要：
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
