arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06209cs.LG
收藏

面向偏微分方程的自监督预训练

Self-supervised Pretraining for Partial Differential Equations

Indian Institute of Technology Kharagpur(印度理工学院卡拉格普尔分校);University of Michigan(密歇根大学);Deep Forest Sciences(深林科学公司);Carnegie Mellon University(卡内基梅隆大学)

Varun Madhavan, Amal S Sebastian, Bharath Ramsundar, Venkatasubramanian Viswanathan

AI总结:

本研究提出一种基于transformer架构的自监督预训练神经PDE求解器,无需重训练即可适配不同PDE参数,可通过少量数据微调提升特定参数性能,且性能随数据与模型规模提升。

摘要:

在本研究中,我们提出了一种构建神经PDE求解器的新方法,该方法利用了基于transformer的神经网络架构的最新进展。我们的模型无需重新训练网络,即可针对不同的PDE参数值给出求解结果。模型训练以自监督方式开展,与语言和视觉任务中使用的预训练方法类似。我们提出假设:该模型实际上是在学习一族(对应多参数的)算子,将初始条件映射到PDE在任意未来时间步t的解。我们将该方法与Fourier Neural Operator(傅里叶神经算子,FNO)进行了对比,结果表明,尽管针对单个参数值的预测误差高于FNO,但该方法能够在PDE参数空间上实现泛化。我们还证实,通过极少量数据对模型进行微调,即可提升其在特定参数上的性能。此外我们还证明,该模型的性能会随数据量以及模型规模的增大而提升。

英文摘要:

In this work, we describe a novel approach to building a neural PDE solver leveraging recent advances in transformer based neural network architectures. Our model can provide solutions for different values of PDE parameters without any need for retraining the network. The training is carried out in a self-supervised manner, similar to pretraining approaches applied in language and vision tasks. We hypothesize that the model is in effect learning a family of operators (for multiple parameters) mapping the initial condition to the solution of the PDE at any future time step t. We compare this approach with the Fourier Neural Operator (FNO), and demonstrate that it can generalize over the space of PDE parameters, despite having a higher prediction error for individual parameter values compared to the FNO. We show that performance on a specific parameter can be improved by finetuning the model with very small amounts of data. We also demonstrate that the model scales with data as well as model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑