One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in $\log n$ precision on contexts of length $n$. We prove that any log-precision transformer can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.
The Role of Logic and Automata in Understanding Transformers
Max-Planck Institute for Software Systems, Kaiserslautern, Germany University of Kaiserslautern-Landau, Kaiserslautern, Germany Institute for Mathematical
NASP-T: A Fuzzy Neuro-Symbolic Transformer for Logic-Constrained Aviation Safety Report Classification
Faculty of Science and Technology (REALTEK)(科学与技术学院);Norwegian University of Life Sciences (NMBU)(挪威生命科学大学);Dresden International University (DIU)(德累斯顿国际大学)
Quantifying Logical Consistency in Transformers via Query-Key Alignment
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究所);Moscow Institute of Physics and Technology(莫斯科物理技术学院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所(AIRI))
Logical Languages Accepted by Transformer Encoders with Hard Attention
Institute for Mathematical and Computational Engineering, Universidad Católica de Chile(智利天主教大学数学与计算工程研究所);IMFD Chile(智利数学与计算工程研究所);CENIA(智利国家人工智能中心);TU Kaiserslautern(凯泽斯劳滕工业大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Steklov Mathematical Institute of Russian Academy of Sciences(俄罗斯科学院斯捷克洛夫数学研究所)
Pathformer: Recursive Path Query Encoding for Complex Logical Query Answering
School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);Guangdong University of Petrochemical Technology(广东石油化工学院);Jiangmen Polytechnic(江门职业技术学院)