Transformer作为转换器
Transformers as Transducers
AI总结:
本文通过RASP变体将Transformer与有限转换器关联,证明其能表达大类的序列到序列映射,并引入B-RASP[pos]和S-RASP扩展,最后证明掩码平均硬注意力Transformer可模拟S-RASP。
摘要:
我们通过将Transformer与有限转换器相关联来研究其序列到序列的映射能力,并发现它们能够表达出人意料的大类转换。为此,我们使用RASP的变体作为中间表示,RASP是一种旨在帮助人们“像Transformer一样思考”的编程语言。我们将现有的布尔变体B-RASP扩展到序列到序列函数,并证明它恰好计算一阶有理函数(如字符串旋转)。然后,我们引入了两个新的扩展。B-RASP[pos]支持基于位置的计算(如复制字符串的前半部分),并包含所有一阶正则函数。S-RASP增加了前缀和,支持额外的算术运算(如字符串平方),并包含所有一阶多项式正则函数。最后,我们证明掩码平均硬注意力Transformer可以模拟S-RASP。
英文摘要:
We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transductions. We do so using variants of RASP, a programming language designed to help people "think like transformers," as an intermediate representation. We extend the existing Boolean variant B-RASP to sequence-to-sequence functions and show that it computes exactly the first-order rational functions (such as string rotation). Then, we introduce two new extensions. B-RASP[pos] enables calculations on positions (such as copying the first half of a string) and contains all first-order regular functions. S-RASP adds prefix sum, which enables additional arithmetic operations (such as squaring a string) and contains all first-order polyregular functions. Finally, we show that masked average-hard attention transformers can simulate S-RASP.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。