arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.13196cs.LGcs.AI
收藏

作为可编程计算机的环形Transformers

Looped Transformers as Programmable Computers

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Princeton University(普林斯顿大学)

Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, Dimitris Papailiopoulos

摘要:

我们提出了一个框架,使用Transformer网络作为通用计算机编程与特定的权重,并把它们放在一个循环。我们的输入序列充当打孔卡,由指令和数据读/写的内存组成。我们证明了一个恒定数量的编码器层可以模拟基本的计算块,包括嵌入编辑操作,非线性函数,函数调用,程序计数器和条件分支。使用这些构建块,我们模拟一个小型的配置集计算机。这允许我们将迭代算法映射到可以由循环的13层Transformer执行的程序。我们展示了这个Transformer如何根据其输入进行指示,可以模拟基本的计算器,基本的线性代数库和采用反向传播的上下文学习算法。我们的工作突出了注意力机制的多功能性,并表明即使是肤浅的Transformers也可以执行成熟的通用程序。

英文摘要:

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑