arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.14749stat.MLcs.AIcs.LG
收藏

测试集中普遍存在的标签错误使机器学习基准不稳定

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

ChipBrain;MIT(麻省理工学院);Cleanlab;AWS(亚马逊云科技)

Curtis G. Northcutt, Anish Athalye, Jonas Mueller

摘要:

我们在10个最常用的计算机视觉、自然语言和音频数据集的测试集中识别标签错误,随后研究这些标签错误影响基准测试结果的可能性。测试集中的错误数量众多且分布广泛:我们估计10个数据集的平均错误率至少为3.3%,例如标签错误至少占ImageNet验证集的6%。假定的标签错误使用自信的学习算法来识别,然后通过众包进行人工验证(平均而言,在数据集中,51%的算法标记的候选人确实被错误地标记)。传统上,机器学习从业者根据测试准确性选择部署哪个模型-我们的研究结果在这里建议谨慎,建议在正确标记的测试集上判断模型可能更有用,特别是对于嘈杂的真实世界数据集。令人惊讶的是,我们发现,在具有高比例错误标记数据的真实数据集中,低容量模型实际上可能比高容量模型更有用。例如,在具有校正标签的ImageNet上:如果最初错误标记的测试示例的流行率仅增加6%,则ResNet-18优于ResNet-50。在CIFAR-10上,如果最初错误标记的测试示例的患病率仅增加5%,则VGG-11优于VGG-19。可在https://labelerrors.com上查看10个数据集的测试集错误,并可通过https://github.com/cleanlab/label-errors复制所有标签错误。

英文摘要:

We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results. Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set. Putative label errors are identified using confident learning algorithms and then human-validated via crowdsourcing (51% of the algorithmically-flagged candidates are indeed erroneously labeled, on average across the datasets). Traditionally, machine learning practitioners choose which model to deploy based on test accuracy - our findings advise caution here, proposing that judging models over correctly labeled test sets may be more useful, especially for noisy real-world datasets. Surprisingly, we find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data. For example, on ImageNet with corrected labels: ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%. On CIFAR-10 with corrected labels: VGG-11 outperforms VGG-19 if the prevalence of originally mislabeled test examples increases by just 5%. Test set errors across the 10 datasets can be viewed at https://labelerrors.com and all label errors can be reproduced by https://github.com/cleanlab/label-errors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑