发表机构
Conflux Labs Ltd; Michigan State University（Conflux Labs Ltd; 密歇根州立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过群论陪集视角，揭示Transformer和循环网络在状态跟踪任务中学习到的内部表示，连接部分准确性与学习阶段。
AI 中文摘要
状态跟踪需要组合一系列更新，但仅凭准确性并不能揭示模型学到了什么。我们研究了训练用于预测群元素连乘结果的神经网络。我们在Transformer中识别出商解，即模型恢复商类，同时在其成员中几乎均匀地预测。类大小的倒数无需拟合参数即可预测部分准确性，将基于奇偶性的解释扩展到非奇偶商。我们的基线Transformer在前缀重排下的预测变化很小，超出了精确跟踪的边界。我们证明，对于在均匀独立同分布的全群输入下的有限群，随着前缀长度的增长，最优的忽略顺序的精确准确性收敛到阿贝尔化类大小的倒数，与观察到的阿贝尔化平台一致。顺序更新允许更多：任何对子群的右陪集的分割，无论是正规的还是非正规的，都能在顺序更新中存活。在我们对标准Transformer的普查中，每个恢复的陪集分割都来自正规子群，而参数匹配的循环网络在训练过程中经历了正规和非正规右陪集阶段。在$A_5$上，我们识别出循环状态中编码非正规陪集的低维子空间。在三维情况下，陪集均值向量形成近似十二面体，交换这些子空间中的状态分量通过共享输入后缀传递供体的陪集状态。我们的结果通过模型学习跟踪的子群陪集将部分准确性、学习阶段和内部计算联系起来。
英文摘要
State tracking requires composing a sequence of updates, but accuracy alone does not reveal what a model has learned. We study neural networks trained to predict the running product of group elements. We identify quotient solutions in Transformers, where models recover the quotient class while predicting nearly uniformly among its members. The reciprocal of class size predicts partial accuracy without a fitted parameter, extending parity-based accounts to non-parity quotients. Our baseline Transformers' predictions change little under prefix reordering beyond the exact-tracking frontier. We prove that, for finite groups under uniform i.i.d. full-group inputs, optimal order-blind exact accuracy converges to the reciprocal of abelianization class size as prefix length grows, consistent with the observed abelianization plateaus. Sequential updates permit more: any partition into right cosets of a subgroup, normal or not, survives sequential updates. In our census of standard Transformers, every recovered coset partition comes from a normal subgroup, whereas parameter-matched recurrent networks pass through both normal and non-normal right-coset stages during training. On $A_5$, we identify low-dimensional subspaces of the recurrent state that encode non-normal cosets. In the three-dimensional cases, coset mean vectors form approximate dodecahedra, and swapping the state components in these subspaces transfers the donor's coset state through a shared input suffix. Our results connect partial accuracy, learning stages, and internal computation through the subgroup cosets that models learn to track.
Comments69 pages including appendices; 9 pages of main text
