使用图灵程序的通用长度泛化
Universal Length Generalization with Turing Programs
AI总结：
本文提出图灵程序这一通用思维链策略，将算法任务分解为模拟图灵机的步骤，在加法、乘法和上下文SGD等任务上实现Transformer的稳健长度泛化，并从理论上证明Transformer可模拟任意图灵机。
摘要：
长度泛化是指从短训练序列外推到长测试序列的能力，这对当前的大型语言模型来说是一个挑战。虽然先前的工作提出了一些架构或数据格式的改变来实现长度泛化，但这些提议通常只适用于有限的任务集。基于先前的草稿纸（scratchpad）和思维链（Chain-of-Thought, CoT）技术，我们提出了图灵程序（Turing Programs），这是一种新颖的思维链策略，它将算法任务分解为模拟图灵机（Turing Machine）计算的步骤。该框架既具有通用性，因为它可以适应任何算法任务，又具有简单性，只需要从上下文中复制文本并进行少量修改。我们表明，通过使用图灵程序，我们在一系列算法任务上获得了稳健的长度泛化：加法、乘法和上下文随机梯度下降（in-context SGD）。然后我们证明，Transformer 在随机图灵程序上实现了长度泛化，这表明对于任何算法任务，长度泛化都是可能的。最后，我们从理论上证明 Transformer 可以实现图灵程序，构造了一个简单的 RASP（Weiss 等人）程序来模拟任意图灵机。
英文摘要：
Length generalization refers to the ability to extrapolate from short training sequences to long test sequences and is a challenge for current large language models. While prior work has proposed some architecture or data format changes to achieve length generalization, these proposals typically apply to a limited set of tasks. Building on prior scratchpad and Chain-of-Thought (CoT) techniques, we propose Turing Programs, a novel CoT strategy that decomposes an algorithmic task into steps mimicking the computation of a Turing Machine. This framework is both universal, as it can accommodate any algorithmic task, and simple, requiring only copying text from the context with small modifications. We show that by using Turing Programs, we obtain robust length generalization on a range of algorithmic tasks: addition, multiplication and in-context SGD. We then demonstrate that transformers achieve length generalization on random Turing Programs, suggesting that length generalization is possible for any algorithmic task. Finally, we theoretically prove that transformers can implement Turing Programs, constructing a simple RASP (Weiss et al.) program that simulates an arbitrary Turing machine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
