arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2201.11939cs.LG
收藏

距离越远,性能越差:从图层利用率和模型泛化的角度

With Greater Distance Comes Worse Performance: On the Perspective of Layer Utilization and Model Generalization

CyCraft AI Lab(CyCraft AI实验室)

James Wang, Cheng-Lin Yang

摘要:

深度神经网络的泛化仍然是机器学习中的主要开放问题之一。以前的理论工作集中在推导模型复杂性的严格界限,而经验工作表明,神经网络表现出双下降的训练样本计数和神经网络的大小。在本文中,我们实证研究了神经网络的不同层如何对模型做出不同的贡献;我们发现,早期层通常会学习与训练数据和测试数据的性能相关的表示。实际上,更深的层只能最大限度地降低训练风险,并且无法很好地推广测试或错误标记的数据。我们进一步说明训练的权重到最终层的初始值的距离与泛化误差具有高度相关性,并且可以作为模型过拟合的指标。此外,我们通过重新初始化最终层的权重来支持训练后正则化。我们的研究结果提供了一种有效的方法来估计神经网络的泛化能力,这些定量结果的洞察力可能会启发推导出更好的泛化范围,考虑到神经网络的内部结构。

英文摘要:

Generalization of deep neural networks remains one of the main open problems in machine learning. Previous theoretical works focused on deriving tight bounds of model complexity, while empirical works revealed that neural networks exhibit double descent with respect to both training sample counts and the neural network size. In this paper, we empirically examined how different layers of neural networks contribute differently to the model; we found that early layers generally learn representations relevant to performance on both training data and testing data. Contrarily, deeper layers only minimize training risks and fail to generalize well with testing or mislabeled data. We further illustrate the distance of trained weights to its initial value of final layers has high correlation to generalization errors and can serve as an indicator of an overfit of model. Moreover, we show evidence to support post-training regularization by re-initializing weights of final layers. Our findings provide an efficient method to estimate the generalization capability of neural networks, and the insight of those quantitative results may inspire derivation to better generalization bounds that take the internal structure of neural networks into consideration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑