Learning partial differential equations' (PDEs) solution operators is an essential problem in machine learning. However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs' solution. To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators. By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes. Besides, we introduce a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems. The large model capacity of the transformer architecture grants our model the possibility to scale to large datasets and practical problems. We conduct extensive experiments on multiple challenging datasets from different domains and achieve a remarkable improvement compared with alternative methods. Our code and data are publicly available at \url{https://github.com/thu-ml/GNOT}.
National Center for Supercomputing Applications, University of Illinois Urbana-Champaign(国家超级计算中心,伊利诺伊大学厄巴纳-香槟分校);Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign(贝克曼先进科学与技术研究所,伊利诺伊大学厄巴纳-香槟分校);The Grainger College of Engineering, Department of Aerospace Engineering, University of Illinois Urbana-Champaign(格拉inger工程学院,航空航天工程系,伊利诺伊大学厄巴纳-香槟分校);The Grainger College of Engineering, Department of Mechanical Science and Engineering, University of Illinois Urbana-Champaign(格拉inger工程学院,机械科学与工程系,伊利诺伊大学厄巴纳-香槟分校);The Grainger College of Engineering, Department of Civil and Environmental Engineering, University of Illinois Urbana-Champaign(格拉inger工程学院,土木与环境工程系,伊利诺伊大学厄巴纳-香槟分校);Civil and Urban Engineering Department, New York University Abu Dhabi(纽约大学阿布扎比分校土木与城市工程系);Department of Industrial and Manufacturing Systems Engineering, Kansas State University(工业与制造系统工程系,堪萨斯州立大学)
Geometry Aware Operator Transformer as an Efficient and Accurate Neural Surrogate for PDEs on Arbitrary Domains
Seminar for Applied Mathematics, ETH Zurich, Switzerland(应用数学研讨会,苏黎世联邦理工学院,瑞士);ETH AI Center, Zurich, Switzerland(ETH人工智能中心,苏黎世,瑞士);Department of Mechanical and Process Engineering, ETH Zurich, Switzerland(机械与过程工程系,苏黎世联邦理工学院,瑞士);School of Computer Science, CMU, USA(计算机科学学院,卡内基梅隆大学,美国);Centre for Applicable Mathematics, TIFR, India(应用数学中心,印度泰姬·弗里德曼研究所)
Learnable Koopman-Enhanced Transformer-Based Time Series Forecasting with Spectral Control
Max Planck Institute of Geoanthropology(马克斯·普朗克地理人类学研究所);Department of Electrical Engineering and Information Technology, University of Naples(那不勒斯大学电气工程与信息学院)
Temperature Field Reconstruction of Tungsten Monoblock Divertor on EAST using Physics-aware Neural Operator Transformer
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Institute of Plasma Physics, Chinese Academy of Sciences(中国科学院等离子体物理研究所);Tongling University(铜陵学院);College of Physics, Donghua University(东华大学物理学院);Tsinghua University(清华大学)
RETO: A Rotary-Enhanced Transformer Operator for High-Fidelity Prediction of Automotive Aerodynamics
Department of Mechanics and Aerospace Engineering, Southern University of Science and Technology(南方科技大学机械与航空航天工程系);Shenzhen Key Laboratory of Complex Aerospace Flows, Southern University of Science and Technology(深圳复杂航空航天流动重点实验室);Ningbo Key Laboratory of Advanced Manufacturing Simulation, Eastern Institute of Technology(宁波先进制造模拟重点实验室);Shenzhen Tenfong Science and Technology Co., Ltd.(深圳天丰科技有限公司)
DeepKoopFormer: A Koopman Enhanced Transformer Based Architecture for Time Series Forecasting
Helmholtz Center for Environmental Research-UFZ(海德堡环境研究中心-UFZ);Delft Center for Systems and Control(代尔夫特系统与控制中心);School of Engineering, University of Pittsburgh(匹兹堡大学工程学院)
Universal Approximation of Operators with Transformers and Neural Integral Operators
Department of Mathematics and Statistics(数学与统计学系);Idaho State University Physical Science Complex(爱达荷州立大学物理科学中心);S. 8th Ave., Stop 8085(921 S. 8th Ave., Stop 8085);Pocatello, ID 83209, USA
Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning
Southern University of Science and Technology(南方科技大学);Shenzhen International Center for Mathematics(深圳国际数学中心);National Center for Applied Mathematics Shenzhen (NCAMS)(深圳国家应用数学中心)
在本文中,我们首次将Attention Is All You Need中最先进的Transformer中的自注意力应用于与偏微分方程相关的数据驱动算子学习问题。努力放在一起解释的原理,并提高注意力机制的功效。利用Hilbert空间中的算子逼近理论,首次证明了在标度点积注意力中的softmax归一化是充分的,但不是必要的。在没有softmax的情况下,可以证明线性化的Transformer变体的近似能力与逐层的Petrov-Galerkin投影相当,并且估计与序列长度无关。提出了一种新的层归一化方案,该方案模仿Petrov-Galerkin投影,以允许缩放通过注意层传播,这有助于模型在具有未归一化数据的算子学习任务中实现显着的准确性。最后,我们提出了三个操作学习实验,包括粘性Burgers方程,界面达西流,和反界面系数识别问题。新提出的简单的基于注意力的算子学习器Galerkin Transformer,在训练成本和评估精度方面都比其softmax归一化的同类算法有了显着的改进。
Function graph transformers universally approximate operators between function spaces
Doshisha University(大阪市立大学);RIKEN AIP(日本科学技术厅Advanced Institute for Photonics and Electron器件);Rice University(里士满大学);University of Basel(巴塞尔大学);Simons Chair in Computational and Applied Mathematics and Earth Science(Simons计算与应用数学及地球科学主席职位);University of Helsinki(赫尔辛基大学)
Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
University of Science and Technology of China(中国科学技术大学);CAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China(中国科学技术大学)
Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior
Department of Mathematics, University of California, Los Angeles, CA, USA(加州大学洛杉矶分校数学系);Department of Mathematics, University of South Carolina, Columbia, SC, USA(南卡罗来纳大学数学系)
Training-Free Transformer Merging via Sequential Local Operator Alignment
CISPA Helmholtz Center for Information Security(CISPA赫尔姆霍茨信息安全中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Massachusetts Institute of Technology(麻省理工学院)
In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning
School of Mathematics, University of Minnesota(明尼苏达大学数学学院);Department of Mathematics and Statistics, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校数学与统计学系)