基于 Transformer 的模型在学习模拟结构递归方面尚不完善
Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion
AI总结：
本文提出连接结构递归语法、语义与 Transformer 机制的分析框架，揭示模型依赖捷径算法、边缘情况失效，以及 LLM 难以从上下文归纳递归规则和模拟逐步归约。
摘要：
本文研究基于 Transformer 的模型从示例中学习结构递归的能力。递归是自然语言和形式语言中的普遍概念。结构递归是编程语言和形式数学任务的核心；在这些任务中，符号工具目前优于神经模型，例如推断数据类型之间的语义关系以及模拟程序行为。我们提出一个通用框架，将编程语言领域中结构递归的抽象概念与具体的序列建模问题以及学习到的模型行为良好地联系起来。该框架包含一种刻画结构递归一般“语法”的表示，并结合两种不同的框架来理解其“语义”：一种从编程语言视角看更为自然，另一种则有助于把该视角与对底层 Transformer 架构的机制性理解衔接起来。借助我们的框架这一强有力的概念工具，我们识别出不同设置下的各种问题。被训练用于模拟递归计算的模型尚不能完全把握递归，而是拟合捷径算法，因此无法解决训练分布中代表性不足的某些边缘情况。此外，最先进的大型语言模型（LLM）很难从上下文演示中挖掘递归规则。同时，这些 LLM 在模拟递归函数的归约（逐步计算）时，会以值得注意的方式失败。
英文摘要：
This paper investigates the ability of transformer-based models to learn structural recursion from examples. Recursion is a universal concept in both natural and formal languages. Structural recursion is central to the programming language and formal mathematics tasks where symbolic tools currently excel beyond neural models, such as inferring semantic relations between datatypes and emulating program behavior. We introduce a general framework that nicely connects the abstract concepts of structural recursion in the programming language domain to concrete sequence modeling problems and learned models' behavior. The framework includes a representation that captures the general \textit{syntax} of structural recursion, coupled with two different frameworks for understanding their \textit{semantics} -- one that is more natural from a programming languages perspective and one that helps bridge that perspective with a mechanistic understanding of the underlying transformer architecture. With our framework as a powerful conceptual tool, we identify different issues under various set-ups. The models trained to emulate recursive computations cannot fully capture the recursion yet instead fit short-cut algorithms and thus cannot solve certain edge cases that are under-represented in the training distribution. In addition, it is difficult for state-of-the-art large language models (LLMs) to mine recursive rules from in-context demonstrations. Meanwhile, these LLMs fail in interesting ways when emulating reduction (step-wise computation) of the recursive function.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
