Supervised operator learning is an emerging machine learning paradigm with applications to modeling the evolution of spatio-temporal dynamical systems and approximating general black-box relationships between functional data. We propose a novel operator learning method, LOCA (Learning Operators with Coupled Attention), motivated from the recent success of the attention mechanism. In our architecture, the input functions are mapped to a finite set of features which are then averaged with attention weights that depend on the output query locations. By coupling these attention weights together with an integral transform, LOCA is able to explicitly learn correlations in the target output functions, enabling us to approximate nonlinear operators even when the number of output function in the training set measurements is very small. Our formulation is accompanied by rigorous approximation theoretic guarantees on the universal expressiveness of the proposed model. Empirically, we evaluate the performance of LOCA on several operator learning scenarios involving systems governed by ordinary and partial differential equations, as well as a black-box climate prediction problem. Through these scenarios we demonstrate state of the art accuracy, robustness with respect to noisy input data, and a consistently small spread of errors over testing data sets, even for out-of-distribution prediction tasks.
Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学科学国家重点实验室);School of Mathematics and Information Science, Zhongyuan University of Technology(中原工学院数学与信息科学学院)
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
SKL of Processors, Institute of Computing Technology, CAS(处理器国家重点实验室,计算技术研究所,中国科学院);Intelligent Software Research Center, Institute of Software, CAS(智能软件研究中心,软件研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures
National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校国家超级计算应用中心);Department of Mechanical Engineering, Stanford University(斯坦福大学机械工程系)
Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations
University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);International School for Advanced Studies (SISSA)(国际高等研究学院(SISSA));Zhongyuan University of Technology(中原工学院)
Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning
Southern University of Science and Technology(南方科技大学);Shenzhen International Center for Mathematics(深圳国际数学中心);National Center for Applied Mathematics Shenzhen (NCAMS)(深圳国家应用数学中心)
DAMTP University of Cambridge(剑桥大学数学部);Institute for Foundations of Data Science(数据科学基础研究所);Yale University(耶鲁大学);Yau Mathematical Sciences Center Tsinghua University(清华大学姚贝泰数学科学中心)
Self-Attention to Operator Learning-based 3D-IC Thermal Simulation
University of Science and Technology of China(中国科学技术大学);Eastern Institute of Technology(东部技术研究所);ShanghaiTech University(上海科技大学);Tsinghua University(清华大学);BTD Technology(BTD技术);University of Sheffield(谢菲尔德大学);University of California(加州大学)
Functional Attention: From Pairwise Affinities to Functional Correspondences
Technical University of Munich, Germany(慕尼黑技术大学,德国);Munich Center for Machine Learning (MCML), Germany(慕尼黑机器学习中心(MCML),德国);PIXL, Department of Computer Science, University of Oxford, United Kingdom(牛津大学计算机科学系PIXL,英国);ECE, University of Texas at Austin, USA(德克萨斯大学奥斯汀分校电子与计算机工程系,美国)
在本文中,我们首次将Attention Is All You Need中最先进的Transformer中的自注意力应用于与偏微分方程相关的数据驱动算子学习问题。努力放在一起解释的原理,并提高注意力机制的功效。利用Hilbert空间中的算子逼近理论,首次证明了在标度点积注意力中的softmax归一化是充分的,但不是必要的。在没有softmax的情况下,可以证明线性化的Transformer变体的近似能力与逐层的Petrov-Galerkin投影相当,并且估计与序列长度无关。提出了一种新的层归一化方案,该方案模仿Petrov-Galerkin投影,以允许缩放通过注意层传播,这有助于模型在具有未归一化数据的算子学习任务中实现显着的准确性。最后,我们提出了三个操作学习实验,包括粘性Burgers方程,界面达西流,和反界面系数识别问题。新提出的简单的基于注意力的算子学习器Galerkin Transformer,在训练成本和评估精度方面都比其softmax归一化的同类算法有了显着的改进。
Label-Free Finite-Volume-Residual Training of Attention Graph Neural Networks for Coupled Thermo-Fluid Fields
Interdisciplinary Graduate Programme(交叉学科研究生项目);Nanyang Technological University(南洋理工大学);School of Future Technology(未来技术学院);Fuzhou University(福州大学);College of Computing and Data Science(计算与数据科学学院);Yunnan University(云南大学);School of Mechanical and Aerospace Engineering(机械与航空航天工程学院)
Enhancing Solutions for Complex PDEs: Introducing Complementary Convolution and Equivariant Attention in Fourier Neural Operators
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)