Generalization to out-of-distribution (OOD) data is one of the central problems in modern machine learning. Recently, there is a surge of attempts to propose algorithms that mainly build upon the idea of extracting invariant features. Although intuitively reasonable, theoretical understanding of what kind of invariance can guarantee OOD generalization is still limited, and generalization to arbitrary out-of-distribution is clearly impossible. In this work, we take the first step towards rigorous and quantitative definitions of 1) what is OOD; and 2) what does it mean by saying an OOD problem is learnable. We also introduce a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features. Based on these, we prove OOD generalization error bounds. It turns out that OOD generalization largely depends on the expansion function. As recently pointed out by Gulrajani and Lopez-Paz (2020), any OOD learning algorithm without a model selection module is incomplete. Our theory naturally induces a model selection criterion. Extensive experiments on benchmark OOD datasets demonstrate that our model selection criterion has a significant advantage over baselines.
Improved OOD Generalization via Adversarial Training and Pre-training
University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
School of Computing, National University of Singapore(新加坡国立大学计算机学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Department of Computer Science & Technology, Tsinghua University(清华大学计算机科学与技术系)
Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程研究中心,教育部);Tianjin University of Science and Technology(天津科技大学);Beijing Institute of Technology(北京理工大学)
When is invariance useful in an Out-of-Distribution Generalization problem ?
Preferred Networks
分布外(OOD)泛化问题的目标是训练一个能在所有环境下泛化的预测器。该领域的流行方法使用这样的假设,即这种预测器应该是一个\textit{invariant predictor},它捕获在不同环境中保持恒定的机制。虽然这些方法在各种案例研究中取得了实验成功,但这一假设的理论验证仍有很大的空间。本文提出了一组新的理论条件所需的不变预测,以实现OOD最优性。我们的理论不仅适用于非线性情形,而且推广了\citet{rojas2018invariant}中的必要条件。我们还从我们的理论中推导出帧间梯度对齐算法,并在MNIST衍生的基准数据集上以及在\citet{aubinlinear}提出的三个\textit{Invariance Unit Tests}中的两个上证明了其竞争力。
Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shift
Department of Electrical and Computer Engineering(电气与计算机工程系);Florida Institute for National Security (FINS)(佛罗里达国家安全研究所);Applied Artificial Intelligence Group(应用人工智能小组);University of Florida(佛罗里达大学)
What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
Rensselaer Polytechnic Institute(伦斯勒理工学院);Stony Brook University(石溪大学);Michigan State University(密歇根州立大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院);University of Science and Technology of China(中国科学技术大学);IBM Thomas J. Watson Research Center(IBM托马斯·J.沃森研究中心)
Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Department of Electrical, Computer, and Systems Engineering(电气、计算机与系统工程系);Rensselaer Polytechnic Institute(伦塞拉尔理工大学);Department of Electrical and Systems Engineering(电气与系统工程系);University of Pennsylvania(宾夕法尼亚大学)
Information-Theoretic Generalization Bounds of Replay-based Continual Learning
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);Department of Oncology, University of Cambridge(剑桥大学肿瘤学系);China Telecom, China(中国电信);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis
Rensselaer Polytechnic Institute(伦斯勒理工学院);Michigan State University(密歇根州立大学);MIT-IBM Watson AI Lab, IBM Research(麻省理工学院-IBM沃森人工智能实验室(IBM研究院));IBM Research(IBM研究院);University at Buffalo(布法罗大学)
Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学阿利安斯商学院)
A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Laws
National University of Defense Technology(国防科技大学);Wuhan University(武汉大学);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学信息科学与技术学院深圳校区);Hebei University of Technology(河北工业大学)
Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Sampling
Rensselaer Polytechnic Institute(伦斯勒理工学院);Michigan State University(密歇根州立大学);MIT-IBM Watson AI Lab, IBM Research(IBM研究院麻省理工学院-IBM沃森人工智能实验室);IBM Thomas J. Watson Research Center(IBM托马斯·J·沃森研究中心);University at Buffalo(布法罗大学)