深度学习竞赛的事后剖析：辛普森悖论以及规模度量与形状度量的互补作用
Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
摘要：
为了更好地理解最先进的神经网络（NN）模型的良好泛化性能，特别是基于重尾自正则化（HT-SR）理论的ALPHAHAT度量的成功，我们分析了一个公开的模型语料库，用于预测NN的泛化精度。这些模型包括广泛的质量，并使用一系列架构和正则化超参数进行训练。我们将ALPHAHAT分解为两个子组件度量：基于尺度的度量和基于形状的度量。我们确定了什么是辛普森悖论：“规模”指标（来自传统统计学习理论）在总体上表现良好，但是当正则化超参数变化时，在给定深度的数据的子分区上表现不佳;并且其中“形状”度量（来自HT-SR理论）在数据的每个子分区上表现良好，当超参数对于给定深度的模型变化时，但是当聚集具有变化深度的模型时，总体上表现不佳。我们的研究结果强调了当架构和超参数都不同时比较模型的微妙性;隐式尺度与隐式形状参数在理解NN模型质量方面的互补作用;以及需要超越基于泛化理论上限的一刀切度量来描述NN模型的性能。我们的研究结果还进一步阐明了为什么HT-SR理论的ALPHAHAT度量在预测广泛的CV和NLP模型的泛化方面效果如此之好。
英文摘要：
To understand better good generalization performance in state-of-the-art neural network (NN) models, and in particular the success of the ALPHAHAT metric based on Heavy-Tailed Self-Regularization (HT-SR) theory, we analyze of a corpus of models that was made publicly-available for a contest to predict the generalization accuracy of NNs. These models include a wide range of qualities and were trained with a range of architectures and regularization hyperparameters. We break ALPHAHAT into its two subcomponent metrics: a scale-based metric; and a shape-based metric. We identify what amounts to a Simpson's paradox: where "scale" metrics (from traditional statistical learning theory) perform well in aggregate, but can perform poorly on subpartitions of the data of a given depth, when regularization hyperparameters are varied; and where "shape" metrics (from HT-SR theory) perform well on each subpartition of the data, when hyperparameters are varied for models of a given depth, but can perform poorly overall when models with varying depths are aggregated. Our results highlight the subtlety of comparing models when both architectures and hyperparameters are varied; the complementary role of implicit scale versus implicit shape parameters in understanding NN model quality; and the need to go beyond one-size-fits-all metrics based on upper bounds from generalization theory to describe the performance of NN models. Our results also clarify further why the ALPHAHAT metric from HT-SR theory works so well at predicting generalization across a broad range of CV and NLP models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
