发表机构
DePaul University; University of Tübingen; University of Lille; Boston College; University of Notre Dame; Saarland University; Ruhr University Bochum（德保罗大学; 蒂宾根大学; 里尔大学; 波士顿学院; 圣母大学; 萨尔大学; 波鸿鲁尔大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对Transformer的长度泛化问题，建立了正则语言上长度泛化的完整代数刻画，提出多项式时间决策算法，实验验证其能更准确捕捉Transformer的长度泛化行为。
AI 中文摘要
基于Transformer的语言模型有时能泛化到训练时未见过的更长序列，但目前尚缺乏对哪些任务能实现长度泛化的精确刻画；甚至不清楚Transformer能在哪些正则语言上实现长度泛化，而正则语言是一类基础语言。本文的贡献在于，首次建立了Transformer能在哪些正则语言上实现长度泛化的完整刻画，并提供了一个在该语言的句法幺半群规模下运行的多项式时间决策算法。这些结果依赖于对C-RASP中正则语言的有效刻画，C-RASP是一种近期提出的形式体系，用于表达Transformer能实现长度泛化的语言。该刻画颇具挑战性，因为有限半群的Krohn-Rhodes分解理论等经典工具不足以处理C-RASP：其一，Krohn-Rhodes理论的基本构建块——触发器和单群，无法在C-RASP中表达；其二，C-RASP的基本构建块（无界计数）无法由Krohn-Rhodes理论的有限半群表达。因此，正则语言上的长度泛化由经典有限分解理论无法察觉的代数性质控制。我们将经典分解理论从有限半群推广到整数上的无限加法群，从而能通过整数的迭代 wreath 积来刻画C-RASP，并推导出正则语言成员判定的可证多项式时间算法。在广泛的正则语言测试集上开展的实验证实，本文的理论比现有分类更准确地捕捉了Transformer的长度泛化行为。
英文摘要
Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization. It is not even known which regular languages transformers length-generalize on -- and this is a foundational class of languages. Our contributions are to establish the first complete characterization of which regular languages transformers length-generalize on and provide a decision algorithm running in polynomial time in the size of the language's syntactic monoid. These results rely on an effective characterization of the regular languages in C-RASP, a recently-established formalism that expresses which languages transformers length-generalize on. This characterization is challenging because classical tools like Krohn-Rhodes decomposition theory for finite semigroups are insufficient for C-RASP. Firstly, the basic building blocks of Krohn-Rhodes theory -- flip-flop and simple groups -- are not expressible in C-RASP. Secondly, the basic building block of C-RASP (unbounded counting) is not expressible by the finite semigroups of Krohn-Rhodes theory. Thus, length generalization on regular languages is controlled by an algebraic property that is invisible to classical finite decomposition theory. We generalize classical decomposition theory from finite semigroups to the infinite additive group on the integers, allowing us to characterize C-RASP in terms of iterated wreath products of the integers and derive a provable polynomial-time decision algorithm for regular language membership. Experiments across a broad test suite of regular languages confirm that our theory captures transformers' length-generalization behavior more accurately than existing classifications.
Comments54 pages, 12 figures
