梯度饥饿:神经网络的一种学习倾向
Gradient Starvation: A Learning Proclivity in Neural Networks
摘要:
我们确定并形式化了一个基本的梯度下降现象,导致过度参数化神经网络的学习倾向。当交叉熵损失通过仅捕获与任务相关的特征的子集而最小化时,梯度饥饿出现,尽管存在未能被发现的其他预测特征。这一工作为神经网络中这种特征不平衡的出现提供了理论解释。使用动力系统理论中的工具,我们确定了导致这种不平衡的梯度下降期间学习动态的简单属性,并证明考虑到训练数据中的某些统计结构,可以预期这种情况。基于我们提出的形式主义,我们开发了一种新的正则化方法,旨在解耦特征学习动态,提高精度和鲁棒性的梯度饥饿阻碍的情况下,保证。我们用简单和真实世界的分布外(OOD)泛化实验来说明我们的发现。
英文摘要:
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task, despite the presence of other predictive features that fail to be discovered. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks. Using tools from Dynamical Systems theory, we identify simple properties of learning dynamics during gradient descent that lead to this imbalance, and prove that such a situation can be expected given certain statistical structure in training data. Based on our proposed formalism, we develop guarantees for a novel regularization method aimed at decoupling feature learning dynamics, improving accuracy and robustness in cases hindered by gradient starvation. We illustrate our findings with simple and real-world out-of-distribution (OOD) generalization experiments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。