arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.09190cs.LGcs.AI
收藏

理解和测试深度网络对分布外数据的泛化

Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data

Beijing Jiaotong University(北京交通大学)

Rui Hu, Jitao Sang, Jinqiang Wang, Rui Hu, Chaoquan Jiang

摘要:

深度网络模型在分发(ID)数据上表现出色,但在分发(OOD)数据上可能会严重失败。虽然开发方法的重点是提高面向对象的推广,很少有人注意到评估模型处理面向对象数据的能力。本研究致力于分析实验性ID测试存在的问题,并设计面向对象的测试范式,以准确地评估实际性能。我们的分析是基于三种类型的分布偏移,以产生OOD数据的介绍分类。主要意见包括:(1)ID检验既不能反映单个模型的实际性能,也不能在OOD数据下进行不同模型之间的比较。(2)ID测试失败的原因是由相应的分布偏移导致的学习到的边缘和条件虚假相关。在此基础上,我们提出了新的面向对象的测试范式来评估模型对未知数据的泛化能力,并讨论了如何使用面向对象的测试结果来发现模型的错误,以指导模型调试。

英文摘要:

Deep network models perform excellently on In-Distribution (ID) data, but can significantly fail on Out-Of-Distribution (OOD) data. While developing methods focus on improving OOD generalization, few attention has been paid to evaluating the capability of models to handle OOD data. This study is devoted to analyzing the problem of experimental ID test and designing OOD test paradigm to accurately evaluate the practical performance. Our analysis is based on an introduced categorization of three types of distribution shifts to generate OOD data. Main observations include: (1) ID test fails in neither reflecting the actual performance of a single model nor comparing between different models under OOD data. (2) The ID test failure can be ascribed to the learned marginal and conditional spurious correlations resulted from the corresponding distribution shifts. Based on this, we propose novel OOD test paradigms to evaluate the generalization capacity of models to unseen data, and discuss how to use OOD test results to find bugs of models to guide model debugging.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑