arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.02671cs.LGcs.CC
收藏

表达对数精度Transformer的逻辑

A Logic for Expressing Log-Precision Transformers

New York University(纽约大学);Allen Institute for AI(人工智能研究院)

William Merrill, Ashish Sabharwal

AI总结:

本研究证明对数精度Transformer可等价表达为含多数投票量词的一阶逻辑句子,这是其最紧上界和首次逻辑刻画。

摘要:

解释基于Transformer的语言模型推理能力的一种方式是描述它们能在某些输入文本上解析的逻辑规则类型。近期,Chiang等人(2023)表明,有限精度Transformer可以等价地表达为一阶逻辑的一种推广形式。然而,有限精度Transformer是一种较弱的Transformer变体,因为正如我们所展示的,单个头只能关注固定数量的词元,尤其无法表示均匀注意力。由于广泛关注是Transformer的核心能力,我们提出疑问:一个能够进行普遍关注的最小程度更具表达力的模型,是否也能在逻辑中得到刻画?为此,我们分析了在长度为n的上下文上以前向传播以$\log n$精度计算的Transformer。我们证明,任何对数精度Transformer都可以等价地表达为一个一阶逻辑句子,该句子除了标准的全称量词和存在量词外,还可能包含多数投票量词。这是已知的最紧上界,也是对数精度Transformer的首次逻辑刻画。

英文摘要:

One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in $\log n$ precision on contexts of length $n$. We prove that any log-precision transformer can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑