通过中间层的类中心最近简化
Nearest Class-Center Simplification through Intermediate Layers
摘要:
理论深度学习的最新进展已经引入了训练过程中出现的几何属性,超过插值阈值-训练误差达到零。我们探讨了网络中间层的神经崩溃现象,并强调了深网内部最近类中心失配的内部工作原理。我们进一步表明,这些过程都发生在视觉和语言模型架构。最后,我们提出了一个随机可变性简化损失(SVSL),鼓励更好的几何特征在中间层,并提高了训练指标和泛化。
英文摘要:
Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。