arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.25166cs.AI
收藏

训练变换器作为通用计算机

Towards Training Transformers as a Universal Computer

The University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Allen Institute for AI(人工智能研究院)

Ruize Xu, Chenxiao Yang, Yanhong Li, David McAllester

AI总结:

研究展示小型变换器能学习执行简化却计算通用的MicroPy程序,通过PENCIL框架实现高效执行,验证了标准变换器可作为通用计算机的可行性。

摘要:

我们证明小型变换器能学习执行MicroPy程序,该语言虽简化但计算通用。给定过程定义和表达式,变换器利用PENCIL框架预测小步执行,实现高效运行。在随机生成的无意义MicroPy程序上训练后,模型可泛化至各类人类编写程序,包括位复制、翻转、二进制运算及SAT验证与求解。我们注意到训练模型可实现分布外泛化,即能评估新分布程序。由于MicroPy可表达任何计算,本研究提供实证证据表明标准变换器可被训练为通用计算机。

英文摘要:

We demonstrate that a small transformer can learn to execute programs in MicroPy, a simplified yet computationally universal programming language. Given procedure definitions together with an expression to evaluate, the transformer predicts small-step execution using PENCIL scaffolding for space-efficient execution within a bounded context window. After training on randomly generated, semantically arbitrary MicroPy programs, the learned transformer generalizes to held-out human-written programs including bit copying and flipping, binary addition and multiplication, and SAT verification and solving. The trained model exhibits out-of-distribution generalization: it correctly evaluates programs drawn from a substantially different distribution than the one used to generate its training data. Since MicroPy can express any computation, our results provide empirical evidence that a standard transformer can be trained to act as a universal computer when paired with PENCIL scaffolding. Project page: https://www.micropy.org

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑