排斥性自注意力的非平衡相：混沌、注意力凝聚与涌现局域性
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
浏览论文内容
中文总结 AI 辅助
研究最小循环Transformer中排斥性自注意力的非平衡相，揭示翻转分岔导致混沌、注意力凝聚及涌现局域性等集体现象，表明稀疏注意力可维持持续动力学。
中文摘要 AI 辅助
我们研究了一个最小循环Transformer的非平衡动力学，该Transformer包含$N$个归一化token，$Q=K=I$，以及负值映射$V=-I$。基于相似性的注意力选择邻近表示，而负值映射则将token推离所选场。这种反馈可以持续重组表示几何和注意力网络。对于$d=2$，token位于圆上，其中正多边形是精确不动点。随着注意力反馈强度$\gamma$的增加，多边形通过翻转分岔失去稳定性，产生周期二运动、混沌以及簇交换或簇翻转状态。尽管存在这种时间复杂性，但在有限的固定softmax锐度$\beta$下，当$N\to\infty$时注意力仍然保持弥散。注意力凝聚反而在$\beta\sim N^2$的标度区域中出现。在硬路由极限下，排斥性更新放大局部扰动，路由伙伴切换以弹道方式传输这些扰动，在表示空间中产生涌现蝴蝶锥。高维几何提供了另一种局域化途径。对于$d=N\to\infty$，从高斯初始条件出发的模拟为$\beta=O(1)$处的凝聚转变提供了证据，该转变由动态生成的有限重叠间隙驱动。根据$\gamma$的不同，产生的相包括弥散类单纯形态、共识翻转、具有混沌特征的凝聚主动路由以及碎片化簇翻转。这些结果确立了时间活动、注意力凝聚和几何聚类作为不同的集体现象，并表明稀疏注意力可以维持持续动力学而非使其冻结。
英文摘要
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $γ$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $β$. Attention condensation instead emerges in the scaling regime $β\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a condensation transition at $β=O(1)$, driven by dynamically generated finite overlap gaps. Depending on $γ$, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it.
发表机构
- Boston College（波士顿学院）
- Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。
