语义空间的几何：Transformer架构的连续几何框架

中文摘要 AI 辅助

我们提出了一个连续几何框架，将Transformer架构的离散代数运算建模为语义纤维丛$\calE = \calM \times \R^d$上的积分 - 微分方程（IDE）。从单个几何公理出发，即令牌序列形成配备规范测度格的离散1 - 流形，我们将现代Transformer的每个核心组件（RMSNorm、RoPE、Softmax注意力、FFN、残差流、SGD、权重衰减）转化为微分几何、测度论和随机微积分的连贯词汇。所得框架产生了跨越熵最优传输（注意力作为薛定谔桥）和非平衡热力学（SGD作为违反详细平衡的伊藤扩散）的定量预测。我们在五种架构（Qwen3、LLaMA - 3.1、Gemma - 3、GPT - 2、Mistral）上进行了六部分的实验，参数范围从124M到8B。实验观测结果与几何预测在定量上一致，验证了通过连续随机微分几何分析Transformer可为大语言模型的稳定性极限、上下文边界和优化动态提供预测性描述词汇。

英文摘要

We present a continuous geometric framework that models the discrete algebraic operations of the Transformer architecture as an integro-differential equation (IDE) on a semantic fiber bundle $\calE = \calM \times \R^d$. Beginning from a single geometric axiom -- that the token sequence forms a discrete $1$-manifold equipped with a canonical measure lattice -- we translate every core component of the modern Transformer (RMSNorm, RoPE, Softmax Attention, FFN, Residual Stream, SGD, Weight Decay) into a cohesive vocabulary of differential geometry, measure theory, and stochastic calculus. The resulting framework yields quantitative predictions spanning entropic optimal transport (Attention as a Schrödinger bridge) and non-equilibrium thermodynamics (SGD as Itô diffusion violating detailed balance). We conduct a six-part experimental campaign across five architectures (Qwen3, LLaMA\nobreakdash-3.1, Gemma\nobreakdash-3, GPT-2, Mistral) spanning $124$M to $8$B parameters. The empirical observables are quantitatively consistent with the geometric predictions: the $ε^{-1/2}$ Lipschitz scaling calibration at machine precision ($R^2 = 1.000$), the Lie--Trotter operator-splitting torsion, the symmetric ablation instability confirming the Dual-Law of Topological Stability, the $\calO(1/\sqrt{k})$ thermodynamic suppression of Poincaré recurrence on the RoPE torus, the thermodynamic context-limit phase transition, and the Non-Equilibrium Steady State parameter vortex -- verified across two optimizers (AdamW and Pure SGD) to exclude momentum artifacts. The results demonstrate that analyzing Transformers through the lens of continuous stochastic differential geometry provides a predictive descriptive vocabulary for the stability limits, context bounds, and optimization dynamics of Large Language Models.

发表机构

• INFN, Sezione di Cagliari（意大利国家核物理研究所，卡利亚里分部）

机构由 AI 辅助整理，请以论文原文为准。