In real-world material research, machine learning (ML) models are usually expected to predict and discover novel exceptional materials that deviate from the known materials. It is thus a pressing question to provide an objective evaluation of ML model performances in property prediction of out-of-distribution (OOD) materials that are different from the training set distribution. Traditional performance evaluation of materials property prediction models through random splitting of the dataset frequently results in artificially high performance assessments due to the inherent redundancy of typical material datasets. Here we present a comprehensive benchmark study of structure-based graph neural networks (GNNs) for extrapolative OOD materials property prediction. We formulate five different categories of OOD ML problems for three benchmark datasets from the MatBench study. Our extensive experiments show that current state-of-the-art GNN algorithms significantly underperform for the OOD property prediction tasks on average compared to their baselines in the MatBench study, demonstrating a crucial generalization gap in realistic material prediction tasks. We further examine the latent physical spaces of these GNN models and identify the sources of CGCNN, ALIGNN, and DeeperGATGNN's significantly more robust OOD performance than those of the current best models in the MatBench study (coGN and coNGN), and provide insights to improve their performance.
Hong Kong Baptist University(香港浸会大学);The University of Melbourne(墨尔本大学);The University of Sydney(悉尼大学);Nanjing University of Science and Technology(南京理工大学)
Learning to Select Source Domains: Proxy-Rewarded Policy Optimization for Molecular OOD Generalization
Department of Data Science and Artificial Intelligence, Monash University(墨尔本大学数据科学与人工智能系);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Life Sciences and Technology, Tongji University(同济大学生命科学与技术学院)
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
School of Computing, National University of Singapore(新加坡国立大学计算机学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Department of Computer Science & Technology, Tsinghua University(清华大学计算机科学与技术系)
ADMEOOD: Out-of-Distribution Benchmark for Drug Property Prediction
College of Informatics, Huazhong Agricultural University(华中农业大学信息学院);Shenzhen Institute of Nutrition and Health, Huazhong Agricultural University(华中农业大学深圳营养与健康研究院)
Soft causal learning for generalized molecule property prediction: An environment perspective
University of Science and Technology of China(中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院)
Mitigating Overconfidence in Out-of-Distribution Detection by Capturing Extreme Activations
Amsterdam Public Health Research Institute(阿姆斯特丹公共卫生研究院);Amsterdam UMC(阿姆斯特丹大学医学中心);University of Amsterdam(阿姆斯特丹大学);Institute of Logic, Language and Computation(逻辑、语言与计算研究所);Pacmed(帕克梅德公司)
Materials Property Prediction with Uncertainty Quantification: A Benchmark Study
Khoury College of Computer Science, Northeastern University(东北大学 Khoury 计算机学院);Department of Computer Science and Engineering, University of South Carolina(南卡罗来纳大学计算机科学与工程系)
LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Max Planck Institute for Intelligent Systems, Tübingen, Germany(马克斯·普朗克智能系统研究所,图宾根,德国);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);Google DeepMind(谷歌DeepMind)
Identifying Semantic Component for Robust Molecular Property Prediction
Guangdong University of Technology(广东工业大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Peng Cheng Laboratory(鹏城实验室);Carnegie Mellon University(卡内基梅隆大学)
Invariant Learning via Probability of Sufficient and Necessary Causes
University College London(伦敦大学学院);University of Technology Sydney(悉尼科技大学);Hong Kong Baptist University(香港浸会大学);King’s College London(伦敦国王学院);Zhejiang Lab(之江实验室);ByteDance Research(字节跳动研究院);University of Manchester(曼彻斯特大学)
Instructor-inspired Machine Learning for Robust Molecular Property Prediction
Computer Science Department, Stanford University(斯坦福大学计算机科学系);School of Computer Science and Technology, Wuhan University of Science and Technology(武汉科技大学计算机科学与技术学院);School of Engineering, Westlake University(西湖大学工程学院)
What do we learn? Debunking the Myth of Unsupervised Outlier Detection
Technical University of Munich(慕尼黑工业大学);Helmholtz Center Munich(亥姆霍兹慕尼黑中心);Helmholtz AI(亥姆霍兹人工智能研究所);Klinikum Rechts der Isar(伊萨尔河右岸医院);Imperial College London(帝国理工学院);King’s College London(伦敦国王学院)
Learning material synthesis-process-structure-property relationship by data fusion: Bayesian Coregionalization N-Dimensional Piecewise Function Learning
National Institute of Standards and Technology(美国国家标准与技术研究院);University of Maryland(马里兰大学)
Multimodal Foundation Models for Material Property Prediction and Discovery
Massachusetts Institute of Technology(麻省理工学院);John Hopkins University Applied Physics Laboratory(约翰霍普金斯大学应用物理实验室);University of Chicago(芝加哥大学);Technical University of Denmark(丹麦技术大学)
MoMa: A Modular Deep Learning Framework for Material Property Prediction
Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);Department of Chemical Engineering, Tsinghua University(化学工程系,清华大学)
Establishing Deep InfoMax as an effective self-supervised learning methodology in materials informatics
Leverhulme Research Centre for Functional Materials Design(Leverhulme功能材料设计研究中心);University of Liverpool(利物浦大学);Department of Computer Science(计算机科学系);Department of Chemistry(化学系)
本文提出将 Deep InfoMax 作为材料信息学的自监督学习框架,通过最大化晶体点集表示与向量表示间的互信息,在无性质标签的大规模晶体数据上预训练 Site-Net 模型,从而提升小数据量下带隙和形成能预测的性能。
Supervised Pretraining for Material Property Prediction
Lane Department of Computer Science and Electrical Engineering, West Virginia University(兰恩计算机科学与电气工程系,西弗吉尼亚大学);Department of Physics and Astronomy, West Virginia University(物理与天文学系,西弗吉尼亚大学)