发表机构
Sorbonne Center for Artificial Intelligence; Sorbonne Université; Laboratoire de Probabilités, Statistique et Modélisation; CNRS; Inria Paris（索邦人工智能中心; 索邦大学; 概率、统计与建模实验室; 法国国家科学研究中心; 法国国家信息与自动化研究所巴黎分部）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过插入分隔符使Transformer在4x4乘法中准确率从1%提升至89%，揭示进位以环上角度编码且因果用于预测，实现了可学习的进位传播。
AI 中文摘要
要求Transformer在单次前向传播中执行多位数乘法时，往往失败；对预训练语言模型的解释性研究发现，算术是通过输入范围启发式而非显式进位来解决的。我们从头训练小型Llama风格Transformer，在4x4乘法任务上不使用思维链，发现输入格式具有决定性作用：在数字之间插入空格标记，将精确匹配准确率从1%提升至89%。输出位置按进位链顺序学习，其中具有最长依赖关系的中位数字最后被学习。在模型内部，预测每个数字的分隔符标记（其预测槽位）将进位输入编码为残差流中环上的角度；具有更多不同进位值的示例填充了更多的环。基于列和匹配的示例之间进行激活修补表明，该状态在最后一层之前被因果使用：仅修补预测槽位，在最佳模型的第4块之后，对于某一中间列，最多可在84%的情况下转移源进位，而对于其他列，进位首先在相邻答案槽位组装，然后才到达自身。剩余错误几乎总是差一，这与进位或循环数字码上的小误差一致。
英文摘要
Transformers asked to multiply multi-digit numbers in a single forward pass often fail, and interpretability studies of pretrained language models find arithmetic solved by input-range heuristics rather than by an explicit carry. We train small Llama-style transformers from scratch on 4x4 multiplication without chain of thought and find that the input format is decisive: inserting a space token between digits raises exact-match accuracy from 1% to 89%. Output positions are learned in carry-chain order, with the middle digits, which have the longest-range dependencies, learned last. Inside the model, the separator token that predicts each digit (its prediction slot) encodes the carry-in as an angle on a ring in the residual stream; examples with more distinct carry values fill more of the ring. Activation patching between examples matched on the column sum shows that this state is causally used before the last layer: patching the prediction slot alone transfers the source carry in up to 84% of cases after block 4 for one middle column of our best model, while for other columns the carry is first assembled at the neighboring answer slot before reaching its own. Remaining errors are almost always off by one, consistent with a small error on the carry or on the circular digit code.
