线性可分边缘处的Grokking现象
Grokking at the Edge of Linear Separability
AI总结：
本文在二元逻辑分类任务中证明，当训练数据接近线性可分临界点且垂直方向噪声较强时，梯度下降的隐式偏置会导致损失景观中的平坦方向使训练长时间停留在准稳定解，从而自然产生grokking现象，支持grokking出现在插值阈值附近的猜想。
摘要：
我们研究了一个简单二元逻辑分类任务中的grokking现象——即延迟泛化伴随非单调测试损失行为——在该任务中，“记忆”解和“泛化”解可以被严格定义。令人惊讶的是，我们发现即使在这个最小模型中，当问题参数接近临界点时，grokking也会自然出现，并对其机制提供了实证和分析性的见解。具体而言，借助梯度下降的隐式偏置，我们证明当训练数据集从原点近似线性可分且垂直方向存在强噪声时，逻辑回归可以表现出grokking。其根本原因在于，在临界点附近，损失景观中梯度几乎为零的“平坦”方向导致训练动力学在准稳定解附近任意长时间地徘徊，最终才到达全局最小值。最后，我们强调了我们的发现与近期文献之间的相似性，加强了grokking通常出现在插值阈值附近的猜想，这让人联想到物理系统中经常观察到的临界现象。
英文摘要：
We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "memorizing" and "generalizing" solutions can be strictly defined. Surprisingly, we find that grokking arises naturally even in this minimal model when the parameters of the problem are close to a critical point, and provide both empirical and analytical insights into its mechanism. Concretely, by appealing to the implicit bias of gradient descent, we show that logistic regression can exhibit grokking when the training dataset is nearly linearly separable from the origin and there is strong noise in the perpendicular directions. The underlying reason is that near the critical point, "flat" directions in the loss landscape with nearly zero gradient cause training dynamics to linger for arbitrarily long times near quasi-stable solutions before eventually reaching the global minimum. Finally, we highlight similarities between our findings and the recent literature, strengthening the conjecture that grokking generally occurs in proximity to the interpolation threshold, reminiscent of critical phenomena often observed in physical systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
