动力学相位选择控制循环Transformer的计算缩放
Dynamical phase selection controls compute scaling in looped transformers
浏览论文内容
中文总结 AI 辅助
该研究针对循环Transformer，发现架构与准确率相同的网络因初始化不同呈现不同动力学相位，分岔机制决定测试时计算缩放，fold与Neimark–Sacker相位的缩放规律存在差异，表明测试计算由训练解的动力学相位决定。
中文摘要 AI 辅助
循环Transformer通过迭代权重绑定映射执行推理，其计算过程是一个动力学过程，计算成本由推理动力学决定。本文表明，架构和目标相同、训练至相同准确率的网络，会因初始化不同而呈现截然不同的动力学相位，且定义各相位的分岔决定了测试时计算的缩放方式。这些相位通过分岔机制区分，包括鞍结 fold 分岔和 Neimark-Sacker 型向有界非平稳运动的转变。在 fold 相位中，一维范式约化可从训练映射的局部导数预测弛豫时间和谱隙振幅，得到无参数关系 τ(ε)[1−λ_max(−ε)]→π。结合问题难度的正则分布，相同的临界慢化产生工作量水平尾分布 P(τ>N)∼N⁻²。在 Neimark–Sacker 相位中，fold 缩放定律消失而非仅改变其 prefactor。因此，测试时的计算并非仅由架构决定，而是由训练得到的解的动力学相位决定。
英文摘要
A looped transformer performs inference by iterating a weight-tied map, making its computation a dynamical process whose cost is set by the resulting inference dynamics. Here we show that networks with identical architecture and objective, trained to identical accuracy, nevertheless realize distinct dynamical phases depending strongly on initialization, and that the bifurcation defining each phase determines how test-time compute scales. The phases are distinguished by their bifurcation mechanisms, including a saddle-node fold and a Neimark-Sacker-type transition to bounded nonstationary motion. In the fold phase, a one-dimensional normal-form reduction predicts both the relaxation-time and spectral-gap amplitudes from local derivatives of the trained map, yielding the parameter-free relation $τ(\varepsilon)[1-λ_{\max}(-\varepsilon)]\toπ$. Composed with a regular distribution of problem difficulty, the same critical slowing down produces the workload-level tail $P(τ>N)\sim N^{-2}$. In the Neimark--Sacker phase, the fold scaling law disappears rather than merely changing its prefactor. Thus, test-time compute is not determined by architecture alone. It is governed by the dynamical phase of the solution found by training.
发表机构
- Sejong University（世宗大学）
机构由 AI 辅助整理，请以论文原文为准。
