arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.06716cs.LGphysics.data-anstat.ML
收藏

预测最先进的神经网络的质量趋势,而无需访问训练或测试数据

Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data

Calculation Consulting(计算咨询公司);ICSI(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校)

Charles H. Martin, Tongsu, Peng, Michael W. Mahoney

摘要:

在许多应用中,人们使用由其他人训练的神经网络模型。对于这样的预训练模型,可能无法访问训练数据或测试数据。此外,人们可能不知道关于模型的细节,例如,训练数据的细节、损失函数、超参数值等。给定一个或多个预训练模型,要说出模型的预期性能或质量是一个挑战。在这里,我们通过对数百个公开的预训练模型进行详细的元分析来解决这一挑战。我们检查规范为基础的容量控制指标,以及幂律为基础的指标,从最近开发的重尾自正则化理论。我们发现,基于规范的指标与训练良好的模型的测试准确性相关,但它们通常无法区分训练良好的模型和训练不良的模型。我们还发现,基于幂律的度量可以做得更好-定量地更好地区分一系列具有给定架构的训练有素的模型;定性地更好地区分训练有素的模型与训练不足的模型。这些方法可用于识别预训练的神经网络何时存在无法通过检查训练/测试准确性来简单检测的问题。

英文摘要:

In many applications, one works with neural network models trained by someone else. For such pretrained models, one may not have access to training data or test data. Moreover, one may not know details about the model, e.g., the specifics of the training data, the loss function, the hyperparameter values, etc. Given one or many pretrained models, it is a challenge to say anything about the expected performance or quality of the models. Here, we address this challenge by providing a detailed meta-analysis of hundreds of publicly-available pretrained models. We examine norm based capacity control metrics as well as power law based metrics from the recently-developed Theory of Heavy-Tailed Self Regularization. We find that norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly-trained models. We also find that power law based metrics can do much better -- quantitatively better at discriminating among series of well-trained models with a given architecture; and qualitatively better at discriminating well-trained versus poorly-trained models. These methods can be used to identify when a pretrained neural network has problems that cannot be detected simply by examining training/test accuracies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑