arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.03542cs.LGcs.NAmath.NA
收藏

DPOT:面向大规模偏微分方程预训练的自回归去噪算子Transformer

DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training

Tsinghua University(清华大学);Caltech(加州理工学院);RealAI

Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, Jun Zhu

AI总结:

本文提出DPOT,一种基于Fourier attention的自回归去噪PDE预训练Transformer,在10余个数据集、超100k轨迹上以0.5B参数实现SOTA,并显著提升3D等下游任务性能。

摘要:

预训练已被研究用于在数据稀缺场景下提升神经算子训练的效率与性能。然而,由于偏微分方程(PDEs)数据固有的复杂性和多样性,例如长轨迹、多尺度和变化维度,该方向仍处于早期阶段。本文提出了一种新的自回归去噪预训练策略,能够在PDE数据上实现更稳定、更高效的预训练,并可泛化到多种下游任务。此外,通过设计一种基于Fourier attention的灵活且可扩展的模型架构,我们可以轻松扩展模型以进行大规模预训练。我们在10余个PDE数据集、超过100k条轨迹上训练了参数量高达0.5B的PDE基础模型。大量实验表明,我们在这些基准上取得了SOTA结果,并验证了模型强大的泛化能力,能够显著提升多样下游PDE任务(如3D数据)的性能。代码见\url{https://github.com/thu-ml/DPOT}。

英文摘要:

Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑