使用无需访问任何训练或测试数据的泛化指标评估自然语言处理模型
Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data
AI总结：
本文面向NLP任务评估无需训练或测试数据的泛化指标，通过51个Huggingface预训练Transformer和28种指标，发现重尾类指标更能预测测试误差。
摘要：
选择合适的架构参数和训练超参数对于提升机器学习（ML）模型性能至关重要。近期若干实证研究对神经网络（NN）开展大规模相关性分析，以寻找能够指导此类模型选择的有效泛化指标。有效指标通常被期望与测试性能具有强相关性。本文扩展了此前的分析，考察基于泛化指标的模型选择，目标如下：（i）聚焦自然语言处理（NLP）任务，因为先前工作主要集中于计算机视觉（CV）任务；（ii）考虑直接预测测试误差而非泛化差距的指标；（iii）探索计算时无需访问数据的指标。基于这些目标，我们首次利用泛化指标给出了针对来自 Huggingface 的大型预训练 Transformers 的模型选择结果。我们的分析考虑：（I）数百个在不同设置下训练的 Transformers，其中我们系统地改变数据量、模型规模和优化超参数；（II）来自八个 Huggingface NLP 模型家族的共51个预训练 Transformers，包括 GPT2、BERT 等；以及（III）共28个现有和新型泛化指标。尽管源自重尾（HT）视角的指标处于小众地位，但我们发现它们在 NLP 任务中特别有用，表现出比其他更流行指标更强的相关性。为进一步考察这些指标，我们将依赖幂律（PL）谱分布的既有公式扩展到指数（EXP）和指数截断幂律（E-TPL）族。
英文摘要：
Selecting suitable architecture parameters and training hyperparameters is essential for enhancing machine learning (ML) model performance. Several recent empirical studies conduct large-scale correlational analysis on neural networks (NNs) to search for effective \emph{generalization metrics} that can guide this type of model selection. Effective metrics are typically expected to correlate strongly with test performance. In this paper, we expand on prior analyses by examining generalization-metric-based model selection with the following objectives: (i) focusing on natural language processing (NLP) tasks, as prior work primarily concentrates on computer vision (CV) tasks; (ii) considering metrics that directly predict \emph{test error} instead of the \emph{generalization gap}; (iii) exploring metrics that do not need access to data to compute. From these objectives, we are able to provide the first model selection results on large pretrained Transformers from Huggingface using generalization metrics. Our analyses consider (I) hundreds of Transformers trained in different settings, in which we systematically vary the amount of data, the model size and the optimization hyperparameters, (II) a total of 51 pretrained Transformers from eight families of Huggingface NLP models, including GPT2, BERT, etc., and (III) a total of 28 existing and novel generalization metrics. Despite their niche status, we find that metrics derived from the heavy-tail (HT) perspective are particularly useful in NLP tasks, exhibiting stronger correlations than other, more popular metrics. To further examine these metrics, we extend prior formulations relying on power law (PL) spectral distributions to exponential (EXP) and exponentially-truncated power law (E-TPL) families.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
