arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2201.08924cs.LG
收藏

通过中间层的类中心最近简化

Nearest Class-Center Simplification through Intermediate Layers

Tel Aviv University(特拉维夫大学);eBay Research(eBay研究院)

Ido Ben-Shaul, Shai Dekel

摘要:

理论深度学习的最新进展已经引入了训练过程中出现的几何属性,超过插值阈值-训练误差达到零。我们探讨了网络中间层的神经崩溃现象,并强调了深网内部最近类中心失配的内部工作原理。我们进一步表明,这些过程都发生在视觉和语言模型架构。最后,我们提出了一个随机可变性简化损失(SVSL),鼓励更好的几何特征在中间层,并提高了训练指标和泛化。

英文摘要:

Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑