The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.
Element AI（元素AI）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
深度学习的主要科学挑战之一是解释泛化，即，为什么社区现在训练网络以实现小的训练误差的特定方式也会导致来自同一人群的保持数据的小误差。人们普遍认识到，一些最坏情况下的理论--例如那些基于现代神经网络架构所诱导的预测器类的VC维度的理论--无法解释经验性能。大量的工作旨在缩小这一差距，主要是通过制定泛化误差，优化误差和过度风险的界限。然而，当根据经验进行评估时，大多数这些界限在数值上是空洞的。专注于泛化界限，这项工作解决了如何评估这样的界限经验的问题。Jiang et al.（2020）最近描述了一项大规模的实证研究，旨在揭示边界/度量与泛化之间的潜在因果关系。基于他们的研究，我们强调了他们提出的方法可以掩盖失败和成功的泛化措施解释泛化。我们认为，泛化措施，而不是应该在分布鲁棒性的框架内进行评估。
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）；Max Planck ETH Center for Learning Systems（马克斯·普朗克-苏黎世联邦理工学院学习系统中心）
