为什么累积变换能够外推?
Why Do Accumulated Transformations Extrapolate?
AI总结:
本文研究累积正交变换(如Householder反射或SO(2)旋转)在注意力机制中产生长度外推能力的原理,证明其通过有限步后去相干性抑制远距离token,并指出其最终会退化,而旋转值可扩展有效范围。
摘要:
PaTH Attention 表明,用累积的数据依赖的 Householder 反射替换 RoPE 的位置索引旋转,能够实现强大的长度外推,尽管在极端上下文长度下性能会下降。我们探究这是否依赖于 Householder 特有的结构,还是反映了沿源到查询路径的累积变换的一般性质。我们研究了一个更简单的变体,保留 RoPE 的分块对角 SO(2) 旋转,但将位置索引角度替换为累积的 token 依赖角度。它表现出相同的模式:改进的外推,然后在长上下文下退化。我们证明该结果扩展到满足某些正则条件的累积正交变换:它们的乘积在有限步后变得不相干,抑制了对远距离 token 的注意力。查询和键的累积旋转创建了一个与上下文长度无关的有限混合窗口;训练中学到的逐 token 抑制不变地转移到任何评估长度,高维集中产生一个分数差距,抑制远距离 token,而近路径传输保留目标信号。相反,一个下界表明累积旋转最终必须退化:随着远距离集合增长,没有旋转能在没有显式远距离质量控制的情况下保留近信号。对于 SO(2) 旋转,旋转值也会使残余的远距离贡献不相干地组合,从而扩展范围。受控实验支持这些预测:随机累积旋转显著改进了 RoPE 的外推,学习到的 token 依赖旋转在远超出训练上下文的长度上保持接近训练长度的困惑度,并且旋转值比仅旋转查询和键更有帮助。仅旋转模型在极端长度下仍然退化,而 ALiBi 保持长度稳定,这与需要远距离质量控制的结论一致。
英文摘要:
PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths. We ask whether this depends on Householder-specific structure or reflects a general property of accumulated transformations along source-to-query paths. We study a simpler variant keeping RoPE's block-diagonal SO(2) rotations but replacing position-indexed angles with accumulated token-dependent ones. It shows the same pattern: improved extrapolation then degradation at long contexts. We prove the result extends to accumulated orthogonal transformations satisfying certain regularity conditions: their products become incoherent after finitely many steps, suppressing attention to distant tokens. Accumulated rotations of queries and keys create a finite mixing window independent of context length; per-token suppression learned in training transfers unchanged to any evaluation length, and high-dimensional concentration produces a score gap suppressing far tokens while near-route transport preserves the target signal. Conversely, a lower bound shows accumulated rotations must eventually degrade: as the far set grows, no rotations preserve the near signal without explicit far-mass control. For SO(2) rotations, rotating values too makes residual far contributions combine incoherently, extending the range. Controlled experiments support these predictions: random accumulated rotations substantially improve extrapolation over RoPE, learned token-dependent rotations maintain near-training-length perplexity far beyond the training context, and rotating values helps over queries and keys alone. Rotation-only models still degrade at extreme lengths, while ALiBi stays length-stable, consistent with the need for far-mass control.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。