测量Grokking中的共享性
Measuring Sharpness in Grokking
摘要：
神经网络有时会表现出grokking，这是一种在相应的训练集上获得相同性能后，在验证集上获得完美或接近完美性能的现象。在这篇研讨会论文中，我们介绍了一种基于拟合适当函数形式的鲁棒测量Grokking技术。然后，我们使用它来研究在两种设置下训练和验证准确性中过渡的锐度。第一个设置是Levi等人（2023）开发的理论框架，其中闭合形式的表达式很容易获得。第二种设置是一个双层MLP，用于预测比特的奇偶校验，其中grokking由Miller et al.（2023）的隐藏策略引起。我们发现，相对grokking间隙和grokking锐度之间的趋势是相似的，在这两种设置时，使用绝对和相对的措施的清晰度。考虑到这一点，我们朝着解释一些趋势取得了进展，并确定需要进一步研究，以解开各种机制，影响grokking的清晰度。
英文摘要：
Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained on the corresponding training set. In this workshop paper, we introduce a robust technique for measuring grokking, based on fitting an appropriate functional form. We then use this to investigate the sharpness of transitions in training and validation accuracy under two settings. The first setting is the theoretical framework developed by Levi et al. (2023) where closed form expressions are readily accessible. The second setting is a two-layer MLP trained to predict the parity of bits, with grokking induced by the concealment strategy of Miller et al. (2023). We find that trends between relative grokking gap and grokking sharpness are similar in both settings when using absolute and relative measures of sharpness. Reflecting on this, we make progress toward explaining some trends and identify the need for further study to untangle the various mechanisms which influence the sharpness of grokking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
