Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets to improve generalizability and performance. Despite these advances, our understanding of how pretraining affects neural operators is still limited; studies generally propose tailored architectures and datasets that make it challenging to compare or examine different pretraining frameworks. To address this, we compare various pretraining methods without optimizing architecture choices to characterize pretraining dynamics on different models and datasets as well as to understand its scaling and generalization behavior. We find that pretraining is highly dependent on model and dataset choices, but in general transfer learning or physics-based pretraining strategies work best. In addition, pretraining performance can be further improved by using data augmentations. Lastly, pretraining can be additionally beneficial when fine-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution. Through providing insights into pretraining neural operators for physics prediction, we hope to motivate future work in developing and evaluating pretraining methods for PDEs.
Latent Neural Operator Pretraining for Solving Time-Dependent PDEs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
Simon Fraser University(西蒙弗雷泽大学);Lawrence Berkeley National Laboratory(伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所);University of California, Berkeley(加州大学伯克利分校)
Self-Conditioned Denoising for Atomistic Representation Learning
Department of Chemistry, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, 02139, MA, USA(麻省理工学院化学系);Department of Materials Science and Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, 02139, MA, USA(麻省理工学院材料科学与工程系)
Towards Data-Efficient Cross-Device Generalization of Grad-Shafranov Equilibria via Transfer Learning Neural Operator
Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校格兰杰工程学院核、等离子体与放射工程系);Fusion Energy Division, Oak Ridge National Lab(橡树岭国家实验室聚变能源部);National Center for Supercomputing Applications(国家超级计算应用中心);Department of Applied Mechanics, Indian Institute of Technology Delhi(印度理工学院德里分校应用力学系);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(印度理工学院德里分校亚迪人工智能学院)
Data-Efficient Sleep Staging with Synthetic Time Series Pretraining
Department of Medical Engineering and Technomathematics, FH Aachen University of Applied Sciences(弗劳恩霍夫亚琛应用科学大学医学工程与技术数学系);Department of Information and Computing Sciences, Utrecht University(乌得勒支大学信息与计算科学系);Institute for Data-Driven Technologies, FH Aachen University of Applied Sciences(弗劳恩霍夫亚琛应用科学大学数据驱动技术研究所)
Can Computational Reducibility Lead to Transferable Models for Graph Combinatorial Optimization?
Operations Research, Université de Montréal(运营研究,蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Valence Labs(Valence实验室);Dept. of Mathematics, Boise State University(数学系,博伊州立大学)
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of Cambridge(剑桥大学);LAION;Jülich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Jülich (FZJ)(于利希研究中心);Helmholtz Association(亥姆霍兹联合会);Google DeepMind(谷歌DeepMind);Munich Center for ML(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学)