DNN分类模型在相似但不相同数据集上有效性比较研究的原则性评价方案
A Principled Evaluation Protocol for Comparative Investigation of the Effectiveness of DNN Classification Models on Similar-but-non-identical Datasets
摘要:
深度神经网络(DNN)模型越来越多地使用新的复制测试数据集进行评估,这些数据集经过精心创建,与旧的和流行的基准数据集相似。然而,与预期背道而驰的是,DNN分类模型在这些复制测试数据集上显示出显著的、一致的且在很大程度上无法解释的准确性下降。虽然流行的评估方法是通过利用相应测试数据集中可用的所有数据点来评估模型的准确性,但我们认为这样做会阻碍我们充分捕捉DNN模型的行为,并对其准确性抱有现实的期望。因此,我们提出了一个原则性的评估协议,适用于在多个测试数据集上对DNN模型的准确性进行比较研究,利用可以使用不同标准(包括不确定性相关信息)选择的数据点子集。通过使用这个新的评估协议,我们确定了$564$ DNN模型在(1)CIFAR-10和ImageNet数据集以及(2)它们的复制数据集上的准确性。我们的实验结果表明,所观察到的已建立的基准数据集和它们的复制之间的准确性下降始终低于(即,模型在复制测试数据集上的表现更好)发表的作品中报告的准确性下降,这些发表的作品依赖于不利用不确定性相关信息的传统评估方法。
英文摘要:
Deep Neural Network (DNN) models are increasingly evaluated using new replication test datasets, which have been carefully created to be similar to older and popular benchmark datasets. However, running counter to expectations, DNN classification models show significant, consistent, and largely unexplained degradation in accuracy on these replication test datasets. While the popular evaluation approach is to assess the accuracy of a model by making use of all the datapoints available in the respective test datasets, we argue that doing so hinders us from adequately capturing the behavior of DNN models and from having realistic expectations about their accuracy. Therefore, we propose a principled evaluation protocol that is suitable for performing comparative investigations of the accuracy of a DNN model on multiple test datasets, leveraging subsets of datapoints that can be selected using different criteria, including uncertainty-related information. By making use of this new evaluation protocol, we determined the accuracy of $564$ DNN models on both (1) the CIFAR-10 and ImageNet datasets and (2) their replication datasets. Our experimental results indicate that the observed accuracy degradation between established benchmark datasets and their replications is consistently lower (that is, models do perform better on the replication test datasets) than the accuracy degradation reported in published works, with these published works relying on conventional evaluation approaches that do not utilize uncertainty-related information.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。