精确解体积与Transformer中的长度泛化
Exact-Solution Volume and Length Generalization in Transformers
浏览论文内容
中文总结 AI 辅助
本文通过归一化精确解体积分析Transformer在四个任务上的长度泛化能力，揭示误差来源并改进模型，在10倍长度测试下准确率从60%提升至85%。
中文摘要 AI 辅助
关于Transformer表达性的研究展示了Transformer是否能够解决给定任务，但很少表明学习到的解是否能够泛化到更长的输入长度。我们通过归一化精确解体积（NESV）来研究这个问题：即在有界参数区域内，对所有长度为$n$的输入实现精确解的参数比例。对于固定宽度、单层Transformer，且注意力机制按$\log n$缩放，我们为四个任务建立了NESV的渐近界：FIRST（$\Theta(1)$）、MAJORITY（$\Theta(1/(n\log n))$）、INDEX（$\Theta(1/n^3)$）和PARITY（$0$）。这些结果与之前的实证结果一致：精确解体积随输入长度衰减越快，该任务的长度泛化就越困难。深入分析INDEX，我们的体积分析揭示了两个随$n$增长而增大的误差来源。因此，我们研究了一种结构上消除其中一项的Transformer模型，理论上将NESV界改进为$\Theta(n^{-1})$，并在测试长度为训练长度10倍时，实证达到85%的准确率，而原始模型为60%。我们得出结论，体积分析可能是识别长度敏感性具体来源的有用方法，从而为特定任务的模型改进提供见解。
英文摘要
Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solution, if learned, is generalizable to longer input lengths. We study this question through normalized exact-solution volume (NESV): the fraction of a bounded parameter region that achieves an exact solution on every input of length $n$. For fixed-width, single-layer transformers with $\log n$-scaled attention, we establish asymptotic bounds on NESV for four tasks: FIRST ($Θ(1)$), MAJORITY ($Θ(1/(n\log n))$), INDEX ($Θ(1/n^3)$), and PARITY ($0$). These results are consistent with previous empirical results: the faster the exact-solution volume decays with input length, the harder it is to length-generalize on that task. Looking deeper into INDEX, our volume analysis reveals two error sources that grow with $n$. Consequently, we study a transformer model that would structurally eliminate one of the terms, theoretically improving the NESV bound to $Θ(n^{-1})$, and empirically achieving 85% accuracy when tested at $10\times$ the training length, compared with the 60% accuracy of the original model. We conclude that volume analysis may be a useful approach to identify concrete sources of length sensitivity and thus provide insights into task-specific model refinements.
发表机构
- University of Notre Dame（圣母大学）
机构由 AI 辅助整理，请以论文原文为准。
