arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.08473cs.LG
收藏

神经算子的预训练策略

Strategies for Pretraining Neural Operators

Carnegie Mellon University(卡内基梅隆大学)

Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, Amir Barati Farimani

AI总结:

本文在不调整架构的前提下系统比较神经算子的多种PDE预训练策略,发现迁移学习与基于物理的方法通常最佳,且数据增强、稀缺数据微调及相似分布下游任务可进一步提升效果。

摘要:

偏微分方程(PDE)建模的预训练近来展现出良好前景,可跨数据集扩展神经算子,从而提升泛化能力与性能。尽管取得了这些进展,我们对预训练如何影响神经算子的理解仍然有限;相关研究通常提出定制化架构和数据集,这使得比较或考察不同预训练框架颇具挑战。为解决这一问题,我们在不优化架构选择的情况下比较了多种预训练方法,以刻画不同模型和数据集上的预训练动态,并理解其扩展与泛化行为。我们发现,预训练高度依赖于模型和数据集的选择,但总体而言,迁移学习或基于物理的预训练策略效果最佳。此外,使用数据增强可以进一步提升预训练性能。最后,在数据稀缺场景下进行微调,或泛化到与预训练分布相似的下游数据时,预训练还能带来额外收益。通过为面向物理预测的神经算子预训练提供洞见,我们希望激励未来在PDE预训练方法开发与评估方面的工作。

英文摘要:

Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets to improve generalizability and performance. Despite these advances, our understanding of how pretraining affects neural operators is still limited; studies generally propose tailored architectures and datasets that make it challenging to compare or examine different pretraining frameworks. To address this, we compare various pretraining methods without optimizing architecture choices to characterize pretraining dynamics on different models and datasets as well as to understand its scaling and generalization behavior. We find that pretraining is highly dependent on model and dataset choices, but in general transfer learning or physics-based pretraining strategies work best. In addition, pretraining performance can be further improved by using data augmentations. Lastly, pretraining can be additionally beneficial when fine-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution. Through providing insights into pretraining neural operators for physics prediction, we hope to motivate future work in developing and evaluating pretraining methods for PDEs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑