空间归纳头：多维元胞自动机的上下文学习
Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
浏览论文内容
中文总结 AI 辅助
针对多维元胞自动机中上下文学习因序列化破坏空间邻域连续性的问题，提出空间归纳头两层收集-匹配电路，实现贝叶斯计数，实验验证泛化与机制。
中文摘要 AI 辅助
归纳头为序列数据中的上下文学习提供了一种机制性解释，但现有理论大多假设与预测相关的上下文构成一个连续块。在多维数据中，序列化会打破这一假设，将空间邻居分散到标记序列中相距较远的位置。我们研究Transformer如何克服多维随机和确定性元胞自动机中的这一路由问题，其中每条轨迹由未知的局部规则生成，并以扁平化序列呈现，且没有基于坐标的显式空间归纳偏置。我们引入空间归纳头，这是一种两层“收集-匹配”电路，其中第一层重建相关的空间邻域，第二层将得到的配置与较早出现的配置进行匹配。我们给出了收集操作的两种显式实现，并表明空间路由所需的位置维度仅取决于局部邻域和空间维度，而非网格体积或轨迹视界。我们进一步构建了一个实现贝叶斯计数的匹配层。端到端电路对于随机规则可以任意接近贝叶斯后验，对于确定性规则可以精确预测。实验上，训练后的两层Transformer在二维和一维设置中泛化到未见过的规则，在确定性规则上实现近乎完美的滚动预测，在随机规则上实现与贝叶斯最优预测器相差小于0.005纳特的KL散度。注意力模式和逐层探针与预测的收集-匹配计算一致，为训练后的Transformer中的空间归纳提供了机制性证据。
英文摘要
Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block. In multidimensional data, serialization breaks this assumption by scattering spatial neighbors across distant positions in the token sequence. We study how transformers overcome this routing problem in multidimensional stochastic and deterministic cellular automata, where each trajectory is generated by an unknown local rule and presented as a flattened sequence without an explicit coordinate-based spatial inductive bias. We introduce spatial induction heads, two-layer gather-and-match circuits in which the first layer reconstructs the relevant spatial neighborhood and the second matches the resulting configuration against earlier occurrences. We give two explicit realizations of the gather and show that the positional dimension required for spatial routing depends only on the local neighborhood and spatial dimension, not on grid volume or trajectory horizon. We further construct a matching layer which implements Bayesian counting. The end-to-end circuit can approximate the Bayesian posterior arbitrarily closely for stochastic rules and can predict exactly for deterministic rules. Empirically, trained two-layer transformers generalize to unseen rules in one and two dimensional settings, achieving near-perfect deterministic rollouts and less than 0.005 nats KL from the Bayes-optimal predictor on stochastic rules. Attention patterns and layerwise probes align with the predicted gather-and-match computation, providing mechanistic evidence for spatial induction in trained transformers.
发表机构
- Cornell University（康奈尔大学）
机构由 AI 辅助整理，请以论文原文为准。
