筛查足够
Screening Is Enough
AI总结:
本文提出Multiscreen模型,通过引入筛查机制解决标准softmax注意力的局限性,实现绝对查询-键相关性评估,减少参数并提升稳定性与效率。
摘要:
标准softmax注意力的核心限制在于无法提供独立可解释的查询-键相关性度量:注意力分数无界,而注意力权重仅相对于竞争键定义。因此,无关键无法被显式拒绝,且即使没有真正相关的键,也会分配部分注意力。我们引入了Multiscreen,一种围绕称为筛查的机制构建的语言模型架构,实现了绝对查询-键相关性。与重新分配所有键的注意力不同,筛查计算有界的查询-键相似性并应用显式阈值,丢弃无关键并聚合剩余键而不进行全局竞争。在实验中,Multiscreen在比Transformer基线少约30%的参数的情况下实现了可比的验证损失,并在显著更大的学习率下保持稳定。它在训练上下文之外维持稳定的长上下文困惑度,并在上下文长度增加时显示出微小的检索性能退化。最后,Multiscreen在长上下文长度下实现了更低的全上下文前向传递延迟。
英文摘要:
We call query--key relevance absolute when its values lie on a fixed bounded scale, depend on neither competing keys nor sequence length, require no sequence-length-dependent calibration, and can all be zero. To realize this notion, we introduce screening, whose explicit threshold transforms bounded query--key similarities into relevance values, enabling exact rejection, empty selection, and direct inspection on a common scale. In a controlled comparison of 12 attention mechanisms on a matched Transformer backbone, only screening maintains both low long-context perplexity and robust retrieval beyond the training context; notably, it does so without inference-time scaling. Building on screening, we introduce Multiscreen, a language-model architecture composed of parallel gated screening tiles. Multiscreen retains these long-context gains while achieving greater parameter efficiency, stronger general zero-shot downstream performance, lower training cost at larger scales, and lower model-side time to first token than Transformer baselines. We further develop a normalization design that keeps Multiscreen training stable even at a learning rate of $1$ and show that an adapted version likewise stabilizes Transformer at the same learning rate.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。