发表机构
University of Milano-Bicocca; LIX, Ecole Polytechnique（米兰比可卡大学; 巴黎综合理工学院LIX实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究利用谱几何与算子理论分析Transformer注意力，提出标记差分算子以分离汇点效应与路由能力，并开发出基于注意力的不确定性估计器，在长上下文输入上表现最佳。
AI 中文摘要
在这项工作中，我们通过谱几何和算子理论的视角研究Transformer注意力。我们将每个注意力头视为标记序列上函数希尔伯特空间之间的函数映射，并推导出一个标记差分算子，其谱结构控制标记空间信息如何被路由到输出。我们表明，标准欧几里得谱在结构上受汇点（sinks）偏差影响，将质量集中与真正的路由能力混为一谈。通过将标记空间重新置于由注意力诱导的内在概率几何中，标记差分谱将汇点效应与路由能力分离，并提供对头输出维度的谱描述。这产生了一个统一框架，用于分析注意力图，在单一算子理论框架内解释汇点、路由坍缩和输出维度。在实践中，通过将注意力启发式方法建立在谱几何基础上，我们开发了一种新颖的基于注意力的不确定性估计器，它补充了基于概率的分数，在长上下文输入上取得了最大的提升。
英文摘要
In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a Token Difference Operator, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel attention-based uncertainty estimator that complements probability-based scores, with the largest gains on long-context inputs.
CommentsThis paper has been accepted as poster at Neurips 2026
