发表机构
IBM Research（IBM研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出一种仅需280个参数的通用Transformer，将算法任务建模为电路模型，结合特定位置编码与自主终止准则，在布尔运算、模运算等基准测试上实现了完美长度泛化。
AI 中文摘要
学习可泛化的算法计算对神经网络而言仍是一项挑战，这体现在其在组合性和长度泛化基准测试上持续存在的失败。我们提出了一种可证明正确的Transformer参数化方式（针对布尔代数任务仅需280个可学习参数），该参数化方式能够学习并评估任意深度或长度的问题。我们假设输入是完全加括号的、格式良好的表达式。我们的方法将算法任务概念化为嵌入在Transformer中的电路模型，从而能够在单次前向传播中完成深度为1的电路归约。为实现深度泛化，我们引入了一种位置编码，用于跟踪每个门在电路中的深度，使模型能够通过掩码硬注意力在每次迭代中识别可评估的子表达式，且通过线性注意力实现每次迭代的复杂度为O(n)。结合自主终止准则，模型会在深度为d的问题经过d次迭代后终止，总复杂度为O(n·d)。我们表明，在浅层问题实例（深度1和深度2）上进行训练可有效恢复可解释的参数，这些参数会“瞬间就位”，从而实现精确的长度泛化。尽管我们证实该构造可完美评估任意长度的布尔表达式（一种通用符号计算），但在其他实验中，我们还证明了该Transformer变体能够在其他常见的长度泛化基准测试（包括模运算和ListOps）上实现完美学习和泛化（准确率达100%）。
英文摘要
Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks. We present a provably correct, transformer parameterization (with only 280 learnable parameters for Boolean algebra tasks) capable of learning and evaluating problems of any depth or length. We assume inputs are fully parenthesized, well-formed expressions. Our approach conceptualizes algorithmic tasks as circuit models embedded in transformers, enabling depth-1 circuit reduction in a single forward pass. To achieve depth generalization, we introduce a positional encoding that tracks each gate's depth within the circuit, enabling the model to identify evaluable subexpressions at each iteration via masked hard attention, with $O(n)$ per-iteration complexity via linear attention. Combined with an autonomous halting criterion, the model terminates after $d$ iterations for problems of depth $d$, yielding $O(n \cdot d)$ total complexity. We show that training on shallow problem instances (depth 1 and depth 2) effectively recovers interpretable parameters that {\em snap} into place, resulting in exact length generalization. Though we establish that our construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments we also demonstrate that our transformer variant can learn and generalize perfectly (100% accuracy) on other common length generalization benchmarks, including modular arithmetic and ListOps.
