The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory, invariance principle-based approaches fail in common classification tasks, where invariant (causal) features capture all the information about the label. Are these failures due to the methods failing to capture the invariance? Or is the invariance principle itself insufficient? To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD. In contrast to the linear regression tasks, we show that for linear classification tasks we need much stronger restrictions on the distribution shifts, or otherwise OOD generalization is impossible. Furthermore, even with appropriate restrictions on distribution shifts in place, we show that the invariance principle alone is insufficient. We prove that a form of the information bottleneck constraint along with invariance helps address key failures when invariant features capture all the information about the label and also retains the existing success when they do not. We propose an approach that incorporates both of these principles and demonstrate its effectiveness in several experiments.
Invariant Graph Learning Meets Information Bottleneck for Out-of-Distribution Generalization
School of Cyber Science and Technology, University of Science and Technology of China（中国科学技术大学网络空间安全学院）；Tencent（腾讯）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）
IDOL: Meeting Diverse Distribution Shifts with Prior Physics for Tropical Cyclone Multi-Task Estimation
Zhejiang University of Technology（浙江工业大学）；Shandong University（山东大学）；Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江省可视信息智能处理重点实验室）
University of Basel（巴塞尔大学）；National Centre for Computational Design and Discovery of Novel Materials MARVEL（新型材料计算设计与发现国家中心（MARVEL））；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
Department of Computing, Imperial College London, UK（伦敦帝国学院计算机系）；Department of Mathematics, Imperial College London, UK（伦敦帝国学院数学系）；London School of Geometry（几何与数论研究所）
Trustworthy Representation Learning via Information Funnels and Bottlenecks
Christian Doppler Laboratory for Dependable Intelligent Systems in Harsh Environments（可信智能系统在恶劣环境中的克里斯蒂安·多普勒实验室）；Graz University of Technology（格拉茨技术大学）；Know Center Research GmbH（Know Center研究有限责任公司）
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
Institute for Communications Engineering, Technical University of Munich（慕尼黑工业大学通信工程研究所）；Signal Processing and Speech Communication Laboratory, Graz University of Technology（格拉茨技术大学信号处理与语音通信实验室）；Know-Center GmbH（Know-Center 有限公司）
