arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.16243cs.LG
收藏

用于训练的混合数据无法确保分布外泛化

Mixture Data for Training Cannot Ensure Out-of-distribution Generalization

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Chongqing Jiaotong University(重庆交通大学);Hong Kong Baptist University(香港浸会大学);National University of Defense Technology(国防科技大学);The University of Sydney(悉尼大学)

Songming Zhang, Yuxiao Luo, Qizhou Wang, Haoang Chi, Xiaofeng Chen, Bo Han, Jinyan Li

AI总结:

本文指出增加混合训练数据无法确保OOD泛化,通过将OOD数据定量定义为位于混合训练数据凸包外的样本,建立了新的泛化误差界限,解释了非单调误差趋势,并启发了一种有效的数据选择策略。

摘要:

深度神经网络在处理分布外(OOD)数据时经常面临泛化问题,且影响因素与其各自影响之间仍存在显著的理论空白。来自分布内数据的文献证据表明,如果用于训练的混合数据规模增加,泛化误差可以缩小。然而,对于OOD样本,这种传统理解不再成立——增加训练数据的规模并不总是导致测试泛化误差的减少。事实上,在各种偏移场景中发现了不同的误差趋势,包括在幂律模式下的下降趋势、先下降后上升,或持续的稳定模式。以前的工作对OOD数据进行定性处理,仅将其视为训练期间未见过的样本,这难以解释复杂的非单调趋势。在这项工作中,我们将OOD数据定量地重新定义为位于混合训练数据凸包之外的数据,并建立了新的泛化误差界限,以更好地理解这些反直觉的观察结果。我们对新风险界限的证明表明,对于凸包内的未见数据,可以保证训练良好模型的有效性;更有趣的是,对于超出此覆盖范围的OOD数据,无法确保泛化,这与我们的观察结果一致。此外,我们尝试了各种OOD技术,以强调我们的结果不仅解释了最近OOD泛化工作中的深刻观察,例如多样数据的重要性以及现有算法对未见偏移的敏感性,而且还启发了一种新颖且有效的数据选择策略。

英文摘要:

Deep neural networks often face generalization problems to handle out-of-distribution (OOD) data, and there remains a notable theoretical gap between the contributing factors and their respective impacts. Literature evidence from in-distribution data has suggested that generalization error can shrink if the size of mixture data for training increases. However, when it comes to OOD samples, this conventional understanding does not hold anymore -- Increasing the size of training data does not always lead to a reduction in the test generalization error. In fact, diverse trends of the errors have been found across various shifting scenarios including those decreasing trends under a power-law pattern, initial declines followed by increases, or continuous stable patterns. Previous work has approached OOD data qualitatively, treating them merely as samples unseen during training, which are hard to explain the complicated non-monotonic trends. In this work, we quantitatively redefine OOD data as those situated outside the convex hull of mixed training data and establish novel generalization error bounds to comprehend the counterintuitive observations better. Our proof of the new risk bound agrees that the efficacy of well-trained models can be guaranteed for unseen data within the convex hull; More interestingly, but for OOD data beyond this coverage, the generalization cannot be ensured, which aligns with our observations. Furthermore, we attempted various OOD techniques to underscore that our results not only explain insightful observations in recent OOD generalization work, such as the significance of diverse data and the sensitivity to unseen shifts of existing algorithms, but it also inspires a novel and effective data selection strategy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑