Transformer架构的弯曲时空
The Curved Spacetime of Transformer Architectures
AI总结:
本文提出一个类比广义相对论的几何框架来理解Transformer模型,将注意力解释为离散联络,并通过可视化曲率景观、模拟实验和受控上下文编辑探测偏转,证实了嵌入空间中存在注意力诱导的曲率。
摘要:
我们提出了一个几何框架来理解基于Transformer的语言模型,与广义相对论建立了明确的类比。查询和键在表示空间中诱导出有效度量,注意力则作为一种离散联络,实现值向量在token间的平行移动。堆叠的层提供了离散的时间切片,token表示在这个弯曲流形上演化,而反向传播扮演了最小作用量原理的角色,在参数空间中塑造最小化损失的轨迹。如果这一类比正确,token嵌入就不应在特征空间中走直线路径;相反,它们逐层的步长应随着由嵌入空间曲率介导的相互作用而弯曲和重新定向。为检验这一预测,我们设计了实验来揭示曲率的存在及其后果:(i) 我们为一个完整段落可视化了曲率景观,揭示了局部转角如何在不同token和层间变化;(ii) 我们通过模拟表明,锐角/钝角的多余计数和较长的长度与弦长比无法用维度或偶然性解释;(iii) 受爱因斯坦日食实验启发,我们探测了在受控上下文编辑下的偏转,证明了嵌入轨迹中存在可测量的、与语义一致的弯曲,证实了注意力诱导的曲率。
英文摘要:
We present a geometric framework for understanding Transformer-based language models, drawing an explicit analogy to General Relativity. Queries and keys induce an effective metric on representation space, and attention acts as a discrete connection that implements parallel transport of value vectors across tokens. Stacked layers provide discrete time-slices through which token representations evolve on this curved manifold, while backpropagation plays the role of a least-action principle that shapes loss-minimizing trajectories in parameter space. If this analogy is correct, token embeddings should not traverse straight paths in feature space; instead, their layer-wise steps should bend and reorient as interactions mediated by embedding space curvature. To test this prediction, we design experiments that expose both the presence and the consequences of curvature: (i) we visualize a curvature landscape for a full paragraph, revealing how local turning angles vary across tokens and layers; (ii) we show through simulations that excess counts of sharp/flat angles and longer length-to-chord ratios are not explainable by dimensionality or chance; and (iii) inspired by Einstein's eclipse experiment, we probe deflection under controlled context edits, demonstrating measurable, meaning-consistent bends in embedding trajectories that confirm attention-induced curvature.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。