arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.16241cs.CVcs.LGstat.ML
收藏

鲁棒性的多面性:对分布外泛化的批判性分析

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

UC Berkeley(加州大学伯克利分校);UChicago(芝加哥大学);Google(谷歌)

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, Justin Gilmer

摘要:

我们介绍了四个新的真实世界的分布偏移数据集,包括图像风格,图像模糊,地理位置,相机操作等的变化。通过我们的新数据集,我们评估了以前提出的用于提高分布外鲁棒性的方法,并对其进行了测试。我们发现,使用更大的模型和人工数据增强可以提高对现实世界的分布偏移的鲁棒性,与先前工作中的要求相反。我们发现人工鲁棒性基准的改进可以转移到现实世界的分布偏移,与先前工作中的要求相反。由于我们观察到数据增强可以帮助真实世界的分布偏移,我们还引入了一种新的数据增强方法,该方法提高了最新技术水平,并且优于使用1000倍以上标记数据预训练的模型。总的来说,我们发现,一些方法始终有助于纹理和局部图像统计的分布偏移,但这些方法并不有助于其他一些分布偏移,如地理变化。我们的研究结果表明,未来的研究必须同时研究多个分布偏移,因为我们证明,没有评估的方法一致地提高鲁棒性。

英文摘要:

We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑