Lazy vs hasty：深度网络中的线性化基于示例难度影响学习进度
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
AI总结：
本研究对比深度网络中惰性（线性）与特征学习（非线性）机制，发现特征学习模式优先学习简单示例，导致训练顺序化，揭示了网络按示例难度分配资源的机制。
摘要：
在试图对深度神经网络的成功给出理论解释的尝试中，最近的一系列工作识别了一种所谓的“惰性训练”机制，在该机制中，网络可以很好地通过其初始化附近的线性化来近似。在这里，我们研究了惰性（线性）和特征学习（非线性）机制对基于难度划分的示例子群的比较效果。具体来说，我们表明，在特征学习模式下，较容易的示例被赋予更大的权重，导致与较难的示例相比训练更快。换句话说，非线性动力学倾向于将难度递增的示例的学习顺序化。我们通过不同的方式来量化示例难度，包括c-score、标签噪声以及存在易学习的虚假相关性的情况下，来说明这一现象。我们的结果揭示了深度网络如何根据示例难度分配资源的新理解。
英文摘要：
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization. Here we investigate the comparative effect of the lazy (linear) and feature learning (non-linear) regimes on subgroups of examples based on their difficulty. Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones. In other words, the non-linear dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, including c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a new understanding of how deep networks prioritize resources across example difficulty.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
