已掌握模型更擅长反向学习
Grokked Models are Better Unlearners
AI总结：
已掌握模型在反向学习中表现更优，通过模块化表示减少梯度对齐，实现更高效的遗忘与更稳定的更新。
摘要：
已掌握的延迟泛化，即在模型拟合训练数据之后才出现的特性，已被联系到鲁棒性和表示质量。我们询问这种训练制度是否有助于机器反向学习，即在不完全重新训练的情况下移除指定数据的影响。我们比较了在视觉（CNNs/ResNets在CIFAR、SVHN和ImageNet上）和语言（在TOFU风格设置上使用变压器）中，在grokking过渡前和后应用标准反向学习方法的效果。从已掌握的检查点开始始终产生（i）更高效的遗忘（达到目标遗忘水平所需的更新更少），（ii）更小的副作用（保留和测试性能下降更小），以及（iii）更稳定的更新，相对于在相同反向学习算法下提前停止的模型。对特征和曲率的分析进一步表明，过渡后的模型学习了更具模块化的表示，其遗忘和保留子集之间的梯度对齐减少，这有助于选择性遗忘。我们的结果强调，当模型在预或后grokking状态下作为正交杠杆来执行反向学习时，提供了改进现有反向学习方法的实用配方，而无需改变其算法。
英文摘要：
Grokking-delayed generalization that emerges well after a model has fit the training data-has been linked to robustness and representation quality. We ask whether this training regime also helps with machine unlearning, i.e., removing the influence of specified data without full retraining. We compare applying standard unlearning methods before versus after the grokking transition across vision (CNNs/ResNets on CIFAR, SVHN, and ImageNet) and language (a transformer on a TOFU-style setup). Starting from grokked checkpoints consistently yields (i) more efficient forgetting (fewer updates to reach a target forget level), (ii) less collateral damage (smaller drops on retained and test performance), and (iii) more stable updates across seeds, relative to early-stopped counterparts under identical unlearning algorithms. Analyses of features and curvature further suggest that post-grokking models learn more modular representations with reduced gradient alignment between forget and retain subsets, which facilitates selective forgetting. Our results highlight when a model is trained (pre- vs. post-grokking) as an orthogonal lever to how unlearning is performed, providing a practical recipe to improve existing unlearning methods without altering their algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
