Out-of-distribution (OOD) graph generalization are critical for many real-world applications. Existing methods neglect to discard spurious or noisy features of inputs, which are irrelevant to the label. Besides, they mainly conduct instance-level class-invariant graph learning and fail to utilize the structural class relationships between graph instances. In this work, we endeavor to address these issues in a unified framework, dubbed Individual and Structural Graph Information Bottlenecks (IS-GIB). To remove class spurious feature caused by distribution shifts, we propose Individual Graph Information Bottleneck (I-GIB) which discards irrelevant information by minimizing the mutual information between the input graph and its embeddings. To leverage the structural intra- and inter-domain correlations, we propose Structural Graph Information Bottleneck (S-GIB). Specifically for a batch of graphs with multiple domains, S-GIB first computes the pair-wise input-input, embedding-embedding, and label-label correlations. Then it minimizes the mutual information between input graph and embedding pairs while maximizing the mutual information between embedding and label pairs. The critical insight of S-GIB is to simultaneously discard spurious features and learn invariant features from a high-order perspective by maintaining class relationships under multiple distributional shifts. Notably, we unify the proposed I-GIB and S-GIB to form our complementary framework IS-GIB. Extensive experiments conducted on both node- and graph-level tasks consistently demonstrate the superior generalization ability of IS-GIB. The code is available at https://github.com/YangLing0818/GraphOOD.
Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization
South China University of Technology(华南理工大学)
学习不变(因果)特征的分布外(OOD)推广最近引起了广泛的关注,其中不变风险最小化(VRM)是一个值得注意的解决方案。尽管线性回归理论上有希望,但在线性分类问题中使用递归的挑战仍然存在。通过将信息瓶颈(IB)原理引入到机器学习中,IB-SVM方法已经证明了它解决这些挑战的能力。本文从两个方面进一步改进了IB-SVM。首先,我们证明了在IB-EQUIPMENT中使用的不变特征的支持重叠的关键假设对于保证OOD泛化是强有力的,并且在没有这个假设的情况下仍然有可能实现最优解。其次,我们说明了两种故障模式,IB-ESTA(和ESTA)可能会失败的学习不变的功能,并解决这样的故障,我们提出了一个\textit{Counterfactual Supervision-based Information Bottleneck (CSIB)}学习算法,可证明恢复不变的功能。通过要求反事实推理,CSIB即使在从单个环境访问数据时也能工作。在多个数据集上的实验验证了我们的理论结果。
Invariant Graph Learning Meets Information Bottleneck for Out-of-Distribution Generalization
School of Cyber Science and Technology, University of Science and Technology of China(中国科学技术大学网络空间安全学院);Tencent(腾讯);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院)
Università della Svizzera Italiana(瑞士意大利大学);University of Liechtenstein(利赫泰森大学);Politecnico di Torino(托尼诺理工学院);IBM Research(IBM研究院);Scuola Universitaria Professionale della Svizzera Italiana(瑞士意大利专业大学)
PanGuide3D: Cohort-Robust Pancreas Tumor Segmentation via Probabilistic Pancreas Conditioning and a Transformer Bottleneck
Trailridge School(Trailridge学校);Department of Biochemistry, Grand View University(生物化学系,Grand View大学);Department of Computer Science, Iowa State University(计算机科学系,Iowa State大学)
Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程研究中心,教育部);Tianjin University of Science and Technology(天津科技大学);Beijing Institute of Technology(北京理工大学)
Invariant Information Bottleneck for Domain Generalization
Microsoft Research Asia(微软亚洲研究院);Hong Kong University of Science and Technology(香港科技大学);Washington University in St. Louis(圣路易斯华盛顿大学);University of California, Berkeley(加州大学伯克利分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Deep Learning and the Information Bottleneck Principle
The Edmond and Lilly Safra Center for Brain Sciences, Hebrew University of Jerusalem(耶路撒冷希伯来大学埃德蒙与莉莉·萨夫拉脑科学中心);School of Engineering and Computer Science, Hebrew University of Jerusalem(耶路撒冷希伯来大学工程与计算机科学学院)
Rethinking Generalization in Graph Neural Networks: A Structural Complexity Perspective
Institute of Intelligent Information Processing(智能信息处理研究院);Shanxi University(山西大学);Alliance Manchester Business School University of Manchester(曼彻斯特大学曼彻斯特商业学院);University of Manchester(曼彻斯特大学);Department of Mathematics(数学系);Hong Kong Baptist University(香港 Baptist 大学)
Limits of message passing for node classification: How class-bottlenecks restrict signal-to-noise ratio
Department of Mathematics(数学系);Department of Computing(计算系);UKRI Centre for Doctoral Training in AI for Healthcare(英国研究与创新人工智能医疗健康博士培训中心);Imperial College London(帝国理工学院);MIT Institute for Data, Systems, and Society(MIT数据、系统与社会研究所);I-X Centre for AI in Science(I-X人工智能科学中心);EPSRC Centre for the Mathematics of Precision Healthcare(工程与物理科学研究委员会精准医疗数学中心)
DIB-OD: Preserving the Invariant Core for Robust Heterogeneous Graph Adaptation via Decoupled Information Bottleneck and Online Distillation
School of Information Technology and Engineering, Tianjin University of Technology and Education(天津职业技术师范大学信息技术工程学院);School of Computer Science and Technology, Tiangong University(天工大学计算机科学与技术学院);Department of Computer Science at University of Exeter(埃克塞特大学计算机科学系)