There has recently been increasing attention towards developing foundational neural Partial Differential Equation (PDE) solvers and neural operators through large-scale pretraining. However, unlike vision and language models that make use of abundant and inexpensive (unlabeled) data for pretraining, these neural solvers usually rely on simulated PDE data, which can be costly to obtain, especially for high-dimensional PDEs. In this work, we aim to Pretrain neural PDE solvers on Lower Dimensional PDEs (PreLowD) where data collection is the least expensive. We evaluated the effectiveness of this pretraining strategy in similar PDEs in higher dimensions. We use the Factorized Fourier Neural Operator (FFNO) due to having the necessary flexibility to be applied to PDE data of arbitrary spatial dimensions and reuse trained parameters in lower dimensions. In addition, our work sheds light on the effect of the fine-tuning configuration to make the most of this pretraining strategy. Code is available at https://github.com/BaratiLab/PreLowD.
Latent Neural Operator Pretraining for Solving Time-Dependent PDEs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
CompNO: A Novel Foundation Model approach for solving Partial Differential Equations
Mines Paris - PSL University, Centre for Material Forming (CEMEF)(巴黎 Mines - PSL 大学,材料成型中心(CEMEF));Mines Paris - PSL University, Centre for Robotics (CAOR)(巴黎 Mines - PSL 大学,机器人中心(CAOR))
Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
Simon Fraser University(西蒙弗雷泽大学);Lawrence Berkeley National Laboratory(伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所);University of California, Berkeley(加州大学伯克利分校)
Cross-Domain Pretraining for Steady-State Neural CFD Surrogates
Carnegie Mellon University(卡内基梅隆大学);New York University(纽约大学);Princeton University(普林斯顿大学);Polymathic AI;Flatiron Institute, Center for Computational Astrophysics(Flatiron研究所,计算天体物理中心);Flatiron Institute, Center for Computational Mathematics(Flatiron研究所,计算数学中心);Flatiron Institute, Scientific Computing Core(Flatiron研究所,科学计算核心)
BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
Department of Computer Science, George Mason University, USA(乔治·马歇尔大学计算机科学系);IBM T.J. Watson Research Center, USA(IBM T.J. Watson研究部);Department of Statistics, Rice University(里士大学统计系);Department of System Engineering & Operations Research, George Mason University, USA(乔治·马歇尔大学系统工程与运营管理系)
Towards Data-Efficient Cross-Device Generalization of Grad-Shafranov Equilibria via Transfer Learning Neural Operator
Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校格兰杰工程学院核、等离子体与放射工程系);Fusion Energy Division, Oak Ridge National Lab(橡树岭国家实验室聚变能源部);National Center for Supercomputing Applications(国家超级计算应用中心);Department of Applied Mechanics, Indian Institute of Technology Delhi(印度理工学院德里分校应用力学系);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(印度理工学院德里分校亚迪人工智能学院)
Selfie: Self-supervised Pretraining for Image Embedding
Google Brain(谷歌大脑)
我们引入了一种名为Selfie的预训练技术,它代表SELFie监督图像嵌入。Selfie概括了BERT的掩蔽语言建模的概念(Devlin等人,2019)通过利用对比预测编码损失(Oord等人,2018年)。给定输入图像中的屏蔽补丁,我们的方法学习选择正确的补丁,从同一图像中采样的其他“干扰”补丁,以填充屏蔽位置。该分类目标回避了预测目标块的精确像素值的需要。Selfie的预训练架构包括一个卷积块网络来处理补丁,然后是一个注意力池网络来总结未屏蔽补丁的内容,然后再预测屏蔽补丁。在微调过程中,我们重复使用预训练找到的卷积权重。我们在三个基准测试(CIFAR-10,ImageNet 32 x 32和ImageNet 224 x 224)上评估了Selfie,其中包含不同数量的标记数据,从5%到100%的训练集。与相同网络的标准监督训练相比,我们的预训练方法在所有设置中对ResNet-50进行了一致的改进。值得注意的是,在ImageNet 224 x 224上,每个类有60个示例(5%),我们的方法将ResNet-50的平均准确率从35.6%提高到46.7%,绝对准确率提高了11.1个点。我们的预训练方法还通过显着降低不同运行中测试准确度的标准差,提高了ResNet-50的训练稳定性,特别是在低数据状态下。