arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.04649cs.LGstat.ML
收藏

线上的准确性:关于分布外和分布内泛化之间的强相关性

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

University of California, Berkeley(加利福尼亚大学伯克利分校);Stanford University(斯坦福大学);Tel Aviv University(特拉维夫大学);Toyota Research Institute(丰田研究院)

John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, Ludwig Schmidt

摘要:

为了使机器学习系统可靠,我们必须了解它们在不可见的、非分布环境中的性能。在本文中,我们经验表明,分布外的性能是强相关的分布性能为广泛的模型和分布偏移。具体来说,我们展示了CIFAR-10和ImageNet变体的分布内和分布外性能之间的强相关性,这是一个来自YCB对象的合成姿态估计任务,FMoW-WILDS中的卫星图像分类和iWildCam-WILDS中的野生动物分类。强相关性在模型架构、超参数、训练集大小和训练持续时间上保持不变,并且比现有领域自适应理论所期望的更精确。为了完成图片,我们还调查了相关性较弱的情况,例如来自CIFAR-10-C和组织分类数据集Camelyon 17-WILDS的一些合成分布偏移。最后,我们提供了一个基于高斯数据模型的候选理论,该模型显示了由分布偏移引起的数据协方差的变化如何影响所观察到的相关性。

英文摘要:

For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑