arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.04496cs.LG
收藏

分布外泛化的理论框架

Towards a Theoretical Framework of Out-of-Distribution Generalization

Peking University(北京大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, Liwei Wang

摘要:

对分布外数据的泛化是现代机器学习的核心问题之一。最近,有一个浪潮的尝试提出的算法,主要建立在提取不变特征的想法。虽然直观上是合理的,但对什么样的不变性可以保证OOD推广的理论理解仍然有限,并且推广到任意的分布外显然是不可能的。在这项工作中,我们迈出了严格和定量定义的第一步:1)什么是OOD; 2)什么是OOD问题是可学习的。我们还引入了一个新的概念的扩展函数,它的特点是在多大程度上的方差被放大的测试域的训练域,从而给出了一个定量的意义上的不变特征。在此基础上,我们证明了OOD的泛化误差界。结果表明,OOD推广在很大程度上取决于扩展函数。正如Gulrajani和Lopez-Paz(2020)最近指出的那样,任何没有模型选择模块的OOD学习算法都是不完整的。我们的理论自然地归纳出一个模型选择标准。在基准OOD数据集上的大量实验表明,我们的模型选择标准比基线具有显着的优势。

英文摘要:

Generalization to out-of-distribution (OOD) data is one of the central problems in modern machine learning. Recently, there is a surge of attempts to propose algorithms that mainly build upon the idea of extracting invariant features. Although intuitively reasonable, theoretical understanding of what kind of invariance can guarantee OOD generalization is still limited, and generalization to arbitrary out-of-distribution is clearly impossible. In this work, we take the first step towards rigorous and quantitative definitions of 1) what is OOD; and 2) what does it mean by saying an OOD problem is learnable. We also introduce a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features. Based on these, we prove OOD generalization error bounds. It turns out that OOD generalization largely depends on the expansion function. As recently pointed out by Gulrajani and Lopez-Paz (2020), any OOD learning algorithm without a model selection module is incomplete. Our theory naturally induces a model selection criterion. Extensive experiments on benchmark OOD datasets demonstrate that our model selection criterion has a significant advantage over baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑