ID 和 OOD 性能在真实世界数据集上有时呈负相关
ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets
AI总结：
本文表明真实世界数据中ID与OOD性能常呈负相关，理论解释了其成因及以往研究遗漏的原因，并指出仅用ID性能进行模型选择会错过最佳OOD模型。
摘要：
多项研究比较了计算机视觉和 NLP 中模型的分布内（ID）和分布外（OOD）性能。它们报告了频繁的正相关，有些甚至从未观察到表明必要权衡的负相关。负相关模式的可能性对于确定 ID 性能能否作为 OOD 泛化能力的代理指标很重要。本文通过多个数据集表明，ID 和 OOD 性能之间的负相关确实会在真实世界数据中发生——不仅仅在理论上的最坏情况设置中。我们还在理论上解释了这些情况如何甚至能在最小的线性设置中出现，以及为什么过去的研究由于模型选择的偏差而可能错过这些情况。我们的观察得出了一些与当前大量文献中的建议相矛盾的建议。- 高 OOD 性能有时需要权衡 ID 性能。- 仅关注 ID 性能可能无法带来最佳的 OOD 性能。它可能在 OOD 性能上产生递减（最终为负）的回报。- 在这些情况下，使用 ID 性能进行模型选择（一种常见的推荐做法）的 OOD 泛化研究必然会错过性能最佳的模型，使这些研究对一系列现象视而不见。
英文摘要：
Several studies have compared the in-distribution (ID) and out-of-distribution (OOD) performance of models in computer vision and NLP. They report a frequent positive correlation and some surprisingly never even observe an inverse correlation indicative of a necessary trade-off. The possibility of inverse patterns is important to determine whether ID performance can serve as a proxy for OOD generalization capabilities. This paper shows with multiple datasets that inverse correlations between ID and OOD performance do happen in real-world data - not only in theoretical worst-case settings. We also explain theoretically how these cases can arise even in a minimal linear setting, and why past studies could miss such cases due to a biased selection of models. Our observations lead to recommendations that contradict those found in much of the current literature. - High OOD performance sometimes requires trading off ID performance. - Focusing on ID performance alone may not lead to optimal OOD performance. It may produce diminishing (eventually negative) returns in OOD performance. - In these cases, studies on OOD generalization that use ID performance for model selection (a common recommended practice) will necessarily miss the best-performing models, making these studies blind to a whole range of phenomena.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
