Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.
Out-of-Distribution Generalization in Time Series: A Survey
School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算机与人工智能学院，西南交通大学）；Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education（可持续城市智能交通工程研究中心，教育部）；Information Sciences and Technology Department, the Pennsylvania State University（信息科学与技术系，宾夕法尼亚州立大学）
Towards out-of-distribution generalization in large-scale astronomical surveys: robust networks learn similar representations
BITS Pilani（比尔拉科学技术学院）；New York University（纽约大学）；Flatiron Institute（熨斗研究所）；University of the Western Cape（西开普大学）；Kempner Institute for the Study of Natural and Artificial Intelligence（肯普纳自然与人工智能研究所）；Harvard University（哈佛大学）；South African Radio Astronomy Observatory（南非射电天文台）
A Survey of Learning on Small Data: Generalization, Optimization, and Challenge
Jilin University（吉林大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Southeast University（东南大学）；University of Technology Sydney（悉尼科技大学）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；Agency for Science, Technology and Research（新加坡科技研究局）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Cross-Subject Generalization for EEG Decoding: A Survey of Deep Learning Methods
Department of Computer Science, University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校计算机科学系）；Department of Computer Science, Dartmouth College（达特茅斯学院计算机科学系）；School of Computing, University of Georgia（佐治亚大学计算科学学院）；School of Electrical and Computer Engineering, University of Georgia（佐治亚大学电气与计算机工程学院）
Survey on Generalization Theory for Graph Neural Networks
RWTH Aachen University（亚琛工业大学）；Technical University of Munich（慕尼黑工业大学）；Massachusetts Institute of Technology（麻省理工学院）；Technion - Israel Institute of Technology（以色列理工学院）
Nanyang Technological University（南洋理工大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Lab（上海人工智能实验室）；University of Surrey（萨里大学）
College of Computer Science and Engineering, Northeastern University（计算机科学与工程学院，东北大学）；Distributed Systems Group, TU Wien（分布式系统组，TU Wien）；College of Software, Northeastern University（软件学院，东北大学）；College of Information Science and Engineering, Northeastern University（信息科学与工程学院，东北大学）
Generalizing to Unseen Domains: A Survey on Domain Generalization
Microsoft Research Asia（微软亚洲研究院）；School of Data Science, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Institute for Data Science, Tsinghua University（清华大学数据科学研究院）
CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Jilin University（吉林大学）；Griffith University（格里菲斯大学）；The Chinese University of Hong Kong（香港中文大学）
RMIT University（皇家墨尔本理工大学）；University of Technology Sydney（悉尼科技大学）；Griffith University（格里菲斯大学）；Data61, CSIRO（联邦科学与工业研究组织Data61）；Swinburne University of Technology（斯威本科技大学）
Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Survey
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；Chemnitz University of Technology（技术大学，克明茨）；Institute of Logic, Language, and Computation, University of Amsterdam（逻辑、语言与计算研究所，阿姆斯特丹大学）；School of Philosophy, Psychology & Language Sciences, University of Edinburgh（哲学、心理学与语言科学学院，爱丁堡大学）
Generalization from Low- to Moderate-Resolution Spectra with Neural Networks for Stellar Parameter Estimation: A Case Study with DESI
Department of Physics \& Astronomy, The Johns Hopkins University, Baltimore, MD 21218, USA；Department of Astronomy, The Ohio State University, 140 West 18th Avenue, Columbus, OH 43210, USA；Center for Cosmology；AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA；Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA；School of Astronomy；Space Science, University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China；National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100012, People's Republic of China；Department of Information Systems, E\"otv\"os Lor\'；University, Budapest 1117, Hungary；Department of Applied Mathematics \& Statistics, Johns Hopkins University, Baltimore, MD 21218, USA
