Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning. Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time series problems. Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators. Motivated by this, we study transformers in the function space setting. We formulate attention as a map between infinite dimensional function spaces and prove that the attention mechanism as implemented in practice is a Monte Carlo or finite difference approximation of this operator. The function space formulation allows for the design of transformer neural operators, a class of architectures designed to learn mappings between function spaces. In this paper, we state and prove the first universal approximation result for transformer neural operators, using only a slight modification of the architecture implemented in practice. The prohibitive cost of applying the attention operator to functions defined on multi-dimensional domains leads to the need for more efficient attention-based architectures. For this reason we also introduce a function space generalization of the patching strategy from computer vision, and introduce a class of associated neural operators. Numerical results, on an array of operator learning problems, demonstrate the promise of our approaches to function space formulations of attention and their use in neural operators.
Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations
University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);International School for Advanced Studies (SISSA)(国际高等研究学院(SISSA));Zhongyuan University of Technology(中原工学院)
Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学科学国家重点实验室);School of Mathematics and Information Science, Zhongyuan University of Technology(中原工学院数学与信息科学学院)
Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning
Southern University of Science and Technology(南方科技大学);Shenzhen International Center for Mathematics(深圳国际数学中心);National Center for Applied Mathematics Shenzhen (NCAMS)(深圳国家应用数学中心)
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
SKL of Processors, Institute of Computing Technology, CAS(处理器国家重点实验室,计算技术研究所,中国科学院);Intelligent Software Research Center, Institute of Software, CAS(智能软件研究中心,软件研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures
National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校国家超级计算应用中心);Department of Mechanical Engineering, Stanford University(斯坦福大学机械工程系)
Learning Visually Interpretable Oscillator Networks for Soft Continuum Robots from Video
Department of Advanced Interdisciplinary Studies, The University of Tokyo(东京大学先进跨学科研究系);Institute of Assembly Technology and Robotics, Leibniz University Hannover(莱比锡大学汉诺威装配技术与机器人研究所);Research Center for Advanced Science and Technology, The University of Tokyo(东京大学先进科学研究中心)