Grokking 的拓扑特征
Topological Signatures of Grokking
AI总结：
通过拓扑学视角研究Grokking现象，利用持久同调分析嵌入矩阵的点云，发现H1同调的持久性显著增加，揭示了任务的循环结构。持久同调为表示学习提供了统一的几何和拓扑描述，且与泛化而非记忆相关。
摘要：
我们通过拓扑学视角研究Grokking现象。利用从在模运算中使用不同素数训练的模型嵌入矩阵导出的点云，通过持久同调分析，识别出Grokking的清晰且一致的拓扑特征：H1同调的最大和总持久性显著增加。持久图谱揭示了主导的长寿命拓扑特征以及逐渐结构化的次级特征，反映了任务下的循环结构。与现有频谱和几何诊断方法（如傅里叶分析和局部内在维度）相比，持久同调为表示学习提供了统一的几何和拓扑描述，捕捉了局部和全局的多尺度结构。在不同数据域和控制设置下的消融实验表明，这些拓扑转变与泛化而非记忆相关。我们的结果表明，持久同调为分析神经网络在训练过程中如何内部化潜在结构提供了原理化且可解释的框架。
英文摘要：
We study the grokking phenomenon through the lens of topology. Using persistent homology on point clouds derived from the embedding matrices of a range of models trained on modular arithmetic with varying primes, we identify a clear and consistent topological signature of grokking: a sharp increase in both the maximum and total persistence of first homology ($H_1$). Persistence diagrams reveal the emergence of a dominant long-lived topological feature together with increasingly structured secondary features, reflecting the underlying cyclic structure of the task. Compared to existing spectral and geometric diagnostics -- specifically, Fourier analysis and local intrinsic dimension -- persistent homology provides a unified geometric and topological characterization of representation learning, capturing both local and global multi-scale structure. Ablations across data regimes and control settings show that these topological transitions are tied to generalization rather than memorization. Our results suggest that persistent homology offers a principled and interpretable framework for analyzing how neural networks internalize latent structure during training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
