Recent advances on Out-of-Distribution (OoD) generalization reveal the robustness of deep learning models against distribution shifts. However, existing works focus on OoD algorithms, such as invariant risk minimization, domain generalization, or stable learning, without considering the influence of deep model architectures on OoD generalization, which may lead to sub-optimal performance. Neural Architecture Search (NAS) methods search for architecture based on its performance on the training data, which may result in poor generalization for OoD tasks. In this work, we propose robust Neural Architecture Search for OoD generalization (NAS-OoD), which optimizes the architecture with respect to its performance on generated OoD data by gradient descent. Specifically, a data generator is learned to synthesize OoD data by maximizing losses computed by different neural architectures, while the goal for architecture search is to find the optimal architecture parameters that minimize the synthetic OoD data losses. The data generator and the neural architecture are jointly optimized in an end-to-end manner, and the minimax training process effectively discovers robust architectures that generalize well for different distribution shifts. Extensive experimental results show that NAS-OoD achieves superior performance on various OoD generalization benchmarks with deep models having a much fewer number of parameters. In addition, on a real industry dataset, the proposed NAS-OoD method reduces the error rate by more than 70% compared with the state-of-the-art method, demonstrating the proposed method's practicality for real applications.
Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
IBM Research(IBM研究院)
最近,不变风险最小化(Invariant Risk Minimization,ARJovsky et al.)提出了一个很有前途的解决方案,以解决外的分布(OOD)泛化。在Ahuja等人案中,证明了求解一类新的“集合对策”的纳什均衡等价于求解一个新的“集合对策”。在这项工作中,我们扩展了Ahuja等人的框架。线性回归通过投影的整体游戏上的$\ell_{\infty}$球。我们表明,这种预测有助于实现非平凡的OOD保证,尽管没有实现完美的不变性。对于具有混杂因素的线性模型,我们证明了这些博弈的纳什均衡比标准的经验风险最小化(ERM)更接近理想的OOD解,并且我们还提供了可证明收敛于这些纳什均衡的学习算法。所提出的方法与国家的最先进的实证比较表明,在实现面向对象的解决方案,在几个设置涉及反因果变量和混杂因素的一致收益。
Improved OOD Generalization via Adversarial Training and Pre-training
University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization
Shanghai Jiao Tong University(上海交通大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
深度学习在独立同分布(independent and identical distributed,i.i.d.)数据然而,神经网络的性能通常在遇到分布外(OoD)数据时急剧退化,即,当训练和测试数据从不同的分布中采样时。虽然已经提出了过多的算法用于OoD泛化,但我们对用于训练和评估这些算法的数据的理解仍然停滞不前。在这项工作中,我们首先确定和测量两种不同的分布偏移,是普遍存在于各种数据集。接下来,通过大量的实验,我们比较OoD泛化算法在两组基准测试,每个占主导地位的分布偏移,揭示他们的优势,以及对其他移动的限制。总的来说,我们将来自不同研究领域的现有数据集和算法定位为同一个连贯的图像。它可以作为一个立足点,可以诉诸未来的面向对象的泛化研究。我们的代码可在www.example.com上获得。
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
School of Computing, National University of Singapore(新加坡国立大学计算机学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Department of Computer Science & Technology, Tsinghua University(清华大学计算机科学与技术系)
Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection
Institute for Foundations of Data Science, University of Wisconsin, Madison(数据科学基础研究院,威斯康星大学麦迪逊分校);Department of Computer Sciences, University of Wisconsin, Madison(计算机科学系,威斯康星大学麦迪逊分校);Department of Electrical and Computer Engineering, University of Wisconsin, Madison(电气与计算机工程系,威斯康星大学麦迪逊分校)
Mixture Data for Training Cannot Ensure Out-of-distribution Generalization
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Chongqing Jiaotong University(重庆交通大学);Hong Kong Baptist University(香港浸会大学);National University of Defense Technology(国防科技大学);The University of Sydney(悉尼大学)
Out-of-distribution Generalization for Total Variation based Invariant Risk Minimization
College of Cyber Security, Jinan University(暨南大学网络安全学院);Pratt School of Engineering, Duke University(杜克大学普拉特工程学院);Sino-French Engineer School, Beihang University(北京航空航天大学中法工程师学院)
The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Fudan University(复旦大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Tasks
University of British Columbia(不列颠哥伦比亚大学);Google Research(谷歌研究院)
在本文中,我们研究了神经算法推理任务的OOD泛化,其目标是学习算法(例如,排序、广度优先搜索和深度优先搜索)。首先,我们认为,在这种情况下,面向对象的泛化是显着不同的普通OOD设置。例如,在OOD的图像分类,如\emph{accuracy on the line}一般化的一些现象,这里没有观察到,和技术,如数据增强方法没有帮助的假设,许多增强技术往往被违反。其次,我们分析了主要挑战(例如,输入分布偏移、非代表性数据生成和无信息的验证度量),即,CLRS \citep{deepmind2021clrs},包含30个算法推理任务。我们提出了几种解决方案,包括一个简单而有效的修复输入分布偏移和改进的数据生成。最后,我们提出了一个基于注意力的2 WL-图神经网络(GNN)处理器,它补充了消息传递GNN,因此它们的组合在所有算法上平均优于最先进的模型3%。我们的代码可在:\url{https://github.com/smahdavi4/clrs} .
Learning beyond Site Bias for OOD Generalization in Brain Networks
MBZUAI(机器智能研究院);Zhengzhou University(郑州大学);University Hospital Tübingen(图宾根大学医院);The Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学)