跨Transformer深度的残差流几何分析

中文摘要 AI 辅助

我们提出了一种以转换为中心的Transformer残差流几何分析方法。相对位移衡量了表示在连续层之间移动的“距离”，正交Procrustes分析将每个转换分离为刚性旋转和非刚性残差。在六个指令调优模型上，针对代码生成和跨语言翻译任务，这些测量揭示了可重复的深度规律。相对位移强烈依赖于层；通常在早期和后期较大，中间三分之一较平稳；且在每个模型的不同条件下几乎不变。旋转幅度在深度上几乎恒定，而Procrustes残差和角度集中度仍受深度调制，残差在最终转换处达到峰值。在生成过程中，非英语目标在最后一层的位移和残差比英语目标大。我们将这些呈现为描述性几何规律，而非计算量度或因果解释。贡献在于残差流转换的测量框架，以及在此研究设置下深度曲线依赖于模型且在很大程度上受条件稳定的证据。

英文摘要

We propose a transition-centred geometric analysis of transformer residual streams. Relative displacement measures how \emph{far} representations move between consecutive layers, and orthogonal Procrustes analysis separates each transition into a rigid rotation and a non-rigid residual. Across six instruction-tuned models, on code generation and cross-lingual translation, these measurements reveal reproducible depth regularities. Relative displacement is strongly layer-dependent; typically larger early and late, with a quieter middle third; and nearly invariant across conditions within each model. Rotation magnitude is nearly constant across depth, while Procrustes residual and angle concentration remain depth-modulated, with residual peaking at the final transition. During generation, non-English targets show larger final-layer displacement and residual than English targets. We present these as descriptive geometric regularities, not as measures of computational effort or causal explanations. The contribution is a measurement framework for residual-stream transitions and evidence that, in the settings studied here, depth curves are model-dependent and largely condition-stable.

发表机构

• ProRata AI（ProRata人工智能公司）

机构由 AI 辅助整理，请以论文原文为准。