发表机构
Princeton University; University of California Los Angeles（普林斯顿大学; 加利福尼亚大学洛杉矶分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究循环变换器中深度缩放问题，提出DeepLoop方法，通过控制访问对齐系数形式化绑定深度效应，保持特定架构并设置参数，在不同规模GPT风格模型上实验，结果显示稳定循环深度需考虑参数访问的残差缩放规则。
AI 中文摘要
循环变换器通过多次应用紧凑的物理块堆栈来缩放顺序计算，在不增加存储参数的情况下增加展开深度。这种重用改变了残差缩放问题。我们通过由访问对齐系数κ_R控制的一阶扰动界来形式化这种绑定深度效应。由此产生的DeepLoop方法保持了后LN DeepNorm架构，并针对展开深度N设置α=(2N)^(1/2)和β=(8N)^(-1/2)。在GPT-2小模型和GPT-2中模型规模的GPT风格循环语言模型上，当没有物理块被重新访问时，DeepLoop是中性的，一旦激活循环深度，就会提高验证损失和下游准确率。这些结果表明，稳定的循环深度需要考虑参数访问的残差缩放规则，而不仅仅是名义层数。
英文摘要
Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient $κ_R$. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from $1/4$ to $1/2$ as loop count grows at fixed physical depth. The resulting method, \textbf{DeepLoop}, keeps the Post-LN DeepNorm architecture and sets $α=(2N)^{1/2}$ and $β=(8N)^{-1/2}$ for unrolled depth $N$. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.
Comments25 pages
