arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.09659cs.LGcs.AIcs.CVstat.ML
收藏

令人惊讶的线性关系预测深度网络中的测试性能

A Surprising Linear Relationship Predicts Test Performance in Deep Networks

Center for Brains, Minds, and Machines, MIT(麻省理工学院大脑、心智与机器中心);Alphabet (Google) X(Alphabet(谷歌)X)

Qianli Liao, Brando Miranda, Andrzej Banburski, Jack Hidary, Tomaso Poggio

摘要:

假设两个网络在一个数据集上具有相同的训练损失,那么它们什么时候会有截然不同的测试损失和错误呢?更好地理解这个泛化问题可能会改善深度网络的实际应用。在本文中,我们证明了交叉熵损失对于具有相同架构,相同Meta参数和相同训练误差的两个网络,诱导显着不同的泛化性能是非常简单的:可以用不同级别的“损坏”数据预训练网络,或者简单地用不同高斯标准差的权重初始化网络。最近的理论结果过拟合的推论表明,这些影响是由于一个内在的问题,测量测试性能的交叉熵/指数型损失,它可以分解成两个组件都最小化SGD-其中之一是不相关的预期分类性能。然而,如果我们将这部分损失剔除,训练和测试损失之间就会出现线性关系。在这种转换下,经典的泛化边界是惊人的紧:经验/训练损失非常接近预期/测试损失。此外,分类误差和归一化交叉熵损失之间的经验关系似乎近似单调

英文摘要:

Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to induce significantly different generalization performances for two networks that have the same architecture, the same meta parameters and the same training error: one can either pretrain the networks with different levels of "corrupted" data or simply initialize the networks with weights of different Gaussian standard deviations. A corollary of recent theoretical results on overfitting shows that these effects are due to an intrinsic problem of measuring test performance with a cross-entropy/exponential-type loss, which can be decomposed into two components both minimized by SGD -- one of which is not related to expected classification performance. However, if we factor out this component of the loss, a linear relationship emerges between training and test losses. Under this transformation, classical generalization bounds are surprisingly tight: the empirical/training loss is very close to the expected/test loss. Furthermore, the empirical relation between classification error and normalized cross-entropy loss seem to be approximately monotonic

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑