arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11487cs.LG
收藏

注意力汇是softmax变换器中可证明必要的现象:来自触发条件任务的证据

Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Tel Aviv University(特拉维夫大学)

Yuval Ran-Milo

AI总结:

研究证明softmax变换器中注意力汇在某些情况下是必要的,通过触发条件任务验证,并展示非归一化ReLU注意力可避免汇现象。

摘要:

研究证明softmax变换器中注意力汇在某些情况下是必要的,通过触发条件任务验证,并展示非归一化ReLU注意力可避免汇现象。

英文摘要:

Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are they sometimes functionally necessary in softmax Transformers? We prove that, in some settings, it is the latter: computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models. Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state (e.g., when the model needs to ignore the input). We instantiate this with a concrete task: when a designated trigger token appears, the model must return the average of all preceding token representations, and otherwise output zero, a task which mirrors the functionality of attention heads in the wild (Barbero et al., 2025; Guo et al., 2024). We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior. Experiments validate our predictions and demonstrate they extend beyond the theoretically analyzed setting: softmax models develop strong sinks while ReLU attention eliminates them in both single-head and multi-head variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑