发表机构
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究将 Grokking 作为动态探测工具，通过扫描权重衰减脉冲，发现功能选择的 canalization 现象，即剂量有序时间效应在可见泛化前出现，且测试损失障碍收敛时该效应仍持续。
AI 中文摘要
对于过参数化神经网络，存在大量解可同等拟合训练数据，但在未见过的样本上表现差异极大。Grokking 能区分训练拟合与可见泛化，为研究训练过程中这种选择如何形成提供了窗口。我们在该平台期扫描短时长、固定持续时间的权重衰减（WD）脉冲，并测量它们如何改变后续泛化时间。在三个 Grokking 任务中，这些改变在平台期早期无顺序，但后期形成稳定的剂量顺序：更强的 WD 增加会使泛化更早发生，更强的 WD 减少会使泛化更晚发生。这种顺序在所有三个任务的可见泛化前就已出现。与此同时，受扰动与基线泛化检查点之间的测试损失障碍向零收敛，而有序时间效应仍持续存在。我们将这种日益受限的解选择与持续的剂量有序时间敏感性的组合，称为功能选择的 canalization。
英文摘要
For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalization, providing a window for studying how this selection develops during training. We sweep short, fixed-duration weight decay (WD) perturbations across the pre-generalization plateau and measure how they shift later generalization time. Across three grokking tasks, these shifts are unordered early in the plateau but later form a stable dose ordering before visible generalization, with stronger WD increases leading to earlier generalization and stronger WD decreases leading to later generalization. Test-loss barriers between perturbed and baseline generalization checkpoints collapse toward zero while the ordered timing effects persist. A similar response reorganization is observed under $\ell_1$ regularization in the grokking setting of Junior et al. (2025). Drawing on Waddington's developmental landscape as an analogy, we call this combination of increasingly constrained solution selection and persistent dose-ordered timing shifts the canalization of grokking solution selection. Together, our response maps and loss-barrier measurements reveal a dynamical reorganization before visible generalization that is consistent with the theoretical picture of regularization-driven motion along a stable slow manifold (Boursier et al., 2025).
Comments23 pages, 13 figures
