arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.09619stat.MLcs.CVcs.LG
收藏

识别数据集复制中的统计偏差

Identifying Statistical Bias in Dataset Replication

MIT(麻省理工学院);UC Berkeley(加州大学伯克利分校)

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, Aleksander Madry

摘要:

数据集复制是一个有用的工具,用于评估特定基准测试准确性的提高是否与模型可靠概括能力的提高相对应。在这项工作中,我们提出了不直观但重要的方式,其中标准的数据集复制方法引入了统计偏差,使所得的观察结果发生偏差。我们研究了ImageNet v2,这是ImageNet数据集的复制,即使在控制了标准的数据质量人工测量之后,模型的准确性也会显着下降(11-14%)。我们发现,在纠正识别的统计偏差,只有估计的$3.6\% \pm 1.5\%$的原始$11.7\% \pm 1.0\%$准确性下降仍然无法解释。最后,我们提出了识别和避免数据集复制偏差的具体建议。我们研究的代码可在http://github.com/MadryLab/dataset-replication-analysis上公开获得。

英文摘要:

Dataset replication is a useful tool for assessing whether improvements in test accuracy on a specific benchmark correspond to improvements in models' ability to generalize reliably. In this work, we present unintuitive yet significant ways in which standard approaches to dataset replication introduce statistical bias, skewing the resulting observations. We study ImageNet-v2, a replication of the ImageNet dataset on which models exhibit a significant (11-14%) drop in accuracy, even after controlling for a standard human-in-the-loop measure of data quality. We show that after correcting for the identified statistical bias, only an estimated $3.6\% \pm 1.5\%$ of the original $11.7\% \pm 1.0\%$ accuracy drop remains unaccounted for. We conclude with concrete recommendations for recognizing and avoiding bias in dataset replication. Code for our study is publicly available at http://github.com/MadryLab/dataset-replication-analysis .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑