是什么让循环变换器比非递归变换器表现更好
What Makes Looped Transformers Perform Better Than Non-Recursive Ones
AI总结:
本文通过损失景观几何学解释循环变换器在复杂推理任务中的优势,并提出SHIFT框架提升其训练效率。
摘要:
尽管循环变换器(称为循环注意力,Looped-Attn)在复杂推理任务上经常优于标准变换器(称为单注意力,Single-Attn),但这种优势的机制仍缺乏深入研究。在本文中,我们通过损失景观几何学的视角来解释这一现象,这受到其在样本和Hessian层面的动力学差异的实证观察的启发。为了正式化这一观点,我们扩展了河流-山谷景观模型,通过区分U型山谷(平坦)和V型山谷(陡峭)。基于实证观察,我们推测循环注意力的递归架构会诱导一种景观层面的归纳偏置,倾向于河流-山谷。这种归纳偏置表明,由于山谷跳跃,沿河流的损失收敛会更优,并进一步鼓励学习复杂的模式,相比由单注意力诱导的河流-U-山谷。基于这一见解,我们提出了SHIFT(分阶段分层渐进训练框架),一种有原则的训练策略,它加速了循环注意力的训练过程,同时实现了可比的性能。
英文摘要:
While looped transformers (termed as Looped-Attn) often outperform standard transformers (termed as Single-Attn) on complex reasoning tasks, the mechanism for this advantage remains underexplored. In this paper, we explain this phenomenon through the lens of loss landscape geometry, inspired by empirical observations of their distinct dynamics at both sample and Hessian levels. To formalize this, we extend the River-Valley landscape model by distinguishing between U-shaped valleys (flat) and V-shaped valleys (steep). Based on empirical observations, we conjecture that the recursive architecture of Looped-Attn induces a landscape-level inductive bias towards River-V-Valley. This inductive bias suggest a better loss convergence along the river due to valley hopping, and further encourage learning about complex patterns compared to the River-U-Valley induced by Single-Attn. Building on this insight, we propose SHIFT (Staged HIerarchical Framework for Progressive Training), a principled training strategy that accelerates the training process of Looped-Attn while achieving comparable performances.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。