LeMON: Learning to Learn Multi-Operator Networks(学习如何学习多算子网络)
LeMON: Learning to Learn Multi-Operator Networks
AI总结:
本研究提出LeMON多算子学习预微调策略,结合PDE无关元学习与低秩适配,实现小样本/零样本新PDE算子预测,验证了算子族缩放定律的广泛适配潜力。
摘要:
单算子学习需要训练深度神经网络来学习特定算子,而近期的多算子学习研究采用算子嵌入结构,在多个算子的数据上训练单个神经网络。因此,多算子学习能够在一个模型内预测一系列算子。在本研究中,我们提出了用于求解偏微分方程(PDE)的多算子学习预训练与微调策略。一个关键特性是,通过增加预训练中使用的算子族数量,PDE基础模型可以在仅有限样本的情况下微调到涉及新PDE的下游任务,其性能优于单算子神经网络。具体而言,利用来自不同PDE族的数据预训练的多算子学习模型,在仅使用新族中有限数量的算子进行微调后,即可预测未见过的算子,使其能够充当无数据PDE求解器。我们还证明,所提出的训练与微调方法能够在零样本预测(无样本)的情况下预测新算子。此外,我们引入了一种与PDE无关的元学习算法,通过提供更优的参数初始化过程来提升模型对各类PDE的适应性。为满足计算资源有限的应用需求,我们探索了低秩适配方法,该方法在降低计算成本的同时还能提升求解器精度。最后,通过研究关于算子族数量的缩放定律,我们确立并强调了其在PDE求解任务中广泛适配的潜力。
英文摘要:
Single-operator learning involves training a deep neural network to learn a specific operator, whereas recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. Thus, multi-operator learning is capable of predicting a range of operators within one model. In this work, we propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning. One key aspect is that by increasing the number of families of operators used in pretraining, a PDE foundation model can be fine-tuned to downstream tasks involving new PDEs with a limited number of samples, thus outperforming single operator neural networks. Specifically, a multi-operator learning model pre-trained with data from diverse PDE families can predict unseen operators after fine-tuning with only a limited number of operators from the new family, enabling them to serve as a data-free PDE solver. We also show that the proposed training and fine-tuning method is able to predict new operators in zero-shot prediction without samples. Additionally, we introduce a PDE-agnostic meta-learning algorithm to improve the adaptability of the model to various PDEs by providing a better parameter initialization process. To address the needs of applications with limited computing resources, we explore low-rank adaptation methods that reduce computational costs while enhancing solver accuracy. Lastly, by examining the scaling law with respect to the number of operator families, we establish and highlight its potential for broad adaptation in PDE-solving tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。