稀疏注意力中的路由吸收：为什么随机门难以超越
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
AI总结：
研究发现稀疏注意力中的路由吸收现象，表明端到端方法面临参数不对称性的吸收压力，而后处理方法可避免此问题。
摘要：
Transformer能否在训练过程中学习哪些注意力条目是重要的？原则上是的：注意力分布高度集中，一个小的门控网络可以在事后识别出重要的条目，其准确性接近完美。但实践中却几乎不可能。当稀疏注意力进行端到端训练时，模型的Q/K/V投影会适应所施加的任何掩码，吸收路由信号，直到学习到的门控效果几乎不比固定的随机门控更好。我们称之为路由吸收，并在一个受控的3100万参数Transformer中提出了四个独立的证据来证明这一点：（1）可微的软门控无论门是学习还是随机，都会收敛到几乎相同的困惑度（48.73±0.60 vs. 49.83±0.04在3个种子下）；（2）硬top-k门控通过掩码获得恰好零梯度；（3）一个在共适应的Q/K/V上蒸馏的门控在对抗性掩码上具有高F1但部署时却产生灾难性的困惑度（601.6 vs. 48.6在掩码无关的Q/K/V上）；（4）训练期间的随机掩码化无法防止共适应（部署密集模型时78.2 ppl vs. 基线37.3）。我们将路由吸收与专家混合中的相同现象联系起来，其中随机路由匹配学习路由是因为专家适应任何路由器，但证明注意力表现出更严重的结构性形式：共享的Q/K/V参数使跨层补偿路径成为可能，而MoE中没有这些路径，因为专家是自包含的模块。这意味着端到端稀疏注意力方法中使用每查询token级门控的吸收压力与门控和模型之间的参数不对称性成正比，而后处理方法通过将表征学习与稀疏化分离，完全绕过了这一问题。
英文摘要：
Learned gates can approximate sparse attention patterns on frozen transformers, yet provide limited benefit over random gates when trained jointly with the model. We investigate this difference in a controlled 31M-parameter transformer and attribute it to routing absorption: model representations co-adapt to the imposed mask, reducing the incremental benefit of learned routing. Four experiments characterize the phenomenon. Differentiable soft gating yields perplexities of 48.73 plus or minus 0.60 with learned gates and 49.83 plus or minus 0.04 with frozen random gates over three seeds. Hard top-k masking provides no gradient path to the gate scores in the tested implementation. Gates distilled onto co-adapted and dense-trained Q/K/V both achieve high F1 against oracle masks, but hard-mask deployment yields perplexities of 601.6 and 48.6, respectively. Stochastic mask training also leaves a substantial deployment penalty: dense evaluation yields 78.2 perplexity, compared with 37.3 for the dense baseline. Experiments on Qwen3-1.7B show that increasing the number of trainable attention layers reduces the gap between learned and random gates. We relate these results to co-adaptation in Mixture-of-Experts and propose parameter asymmetry between the gate and the model as a contributing mechanism. For the tested per-query, token-level gates, freezing the model provides stable routing targets and enables effective post-hoc sparsification. The results motivate random-routing controls and separate evaluation of routing quality and model adaptation in sparse attention methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
