发表机构
University of Luxembourg; London Institute of Mathematical Sciences（卢森堡大学; 伦敦数学科学研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究扩展Transformer残差动力学理论，引入FFN作为转向场，经多模型实验验证其提升角预测性能，且切向分量对模型质量与输出多样性至关重要，还可指导块级干预的并行化应用。
AI 中文摘要
仅注意力机制的动力学理论将Transformer残差方向建模为在球面上聚集的粒子。我们通过引入前馈网络（FFN）项作为作用于每个token状态的局部转向场，扩展了这一框架。该理论预测，FFN场的切向分量是残差方向空间中运动所必需的，关键残差方向对应非线性投影平衡点，且对易子缺陷决定了有限的注意力-FFN块何时可被并行加性流准确近似。在GPT-2、Pythia、Mistral和Llama模型上，扩展理论相比仅注意力基线提升了一步角预测性能，FFN的贡献从GPT-2到Llama-3-8B逐渐增大。干预实验显示，仅保留FFN的切向分量可保留大部分模型质量，而仅保留径向分量会导致性能崩溃；切向分量还能在聚合压力下保持输出多样性。实际应用中，对易子缺陷小的层可近似并行化，仅损失适度精度，而缺陷大的层性能会快速下降。这些发现支持将FFN层视为定向转向场，其可塑造Transformer残差几何并决定块级干预的可行性。
英文摘要
Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FFN) term as a local steering field acting on each token state. The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space, that critical residual directions correspond to nonlinear projective equilibria, and that a commutator defect determines when a finite attention--FFN block can be accurately approximated by a parallel, additive flow. Across GPT-2, Pythia, Mistral, and Llama models, the extended theory improves one-step angular prediction relative to an attention-only baseline, with the contribution of the FFN increasing from GPT-2 to Llama-3-8B. Intervention experiments show that retaining only the tangential FFN component preserves most model quality, whereas retaining only the radial component causes performance to collapse. The tangential component also preserves output diversity under aggregation pressure. As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss, whereas layers with large defects degrade rapidly. These findings support the interpretation of FFN layers as directional steering fields that shape Transformer residual geometry and govern the feasibility of block-level interventions.
