arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.06486cs.LGcs.NAmath.NA
收藏

连续注意力用于神经算子

Continuum Attention for Neural Operators

California Institute of Technology(加州理工学院);NVIDIA(英伟达);The Broad Institute of MIT and Harvard(MIT和哈佛大学Broad研究所);Basis Research Institute(Basis研究机构)

Edoardo Calvello, Nikola B. Kovachki, Matthew E. Levine, Andrew M. Stuart

AI总结:

本文提出了一种基于连续注意力机制的神经算子设计方法,通过函数空间建模实现了对非局部映射的高效逼近,并在多个算子学习任务中验证了其有效性。

摘要:

变换器,尤其是注意力机制,在机器学习中已经变得无处不在。其在建模非局部、长距离相关性的成功,使其在自然语言处理、计算机视觉和时间序列问题中得到广泛应用。神经算子,将函数空间映射到函数空间,如果它们是通用的,则必然是非线性和非局部的;因此自然会问注意力机制是否可以用于神经算子的设计。受此启发,我们研究了函数空间中的变换器。我们将注意力视为无限维函数空间之间的映射,并证明实际中实现的注意力机制是该算子的蒙特卡洛或有限差分近似。函数空间的 formulation 允许设计变换器神经算子,这是一种旨在学习函数空间之间映射的架构。在本文中,我们陈述并证明了变换器神经算子的第一个通用逼近结果,仅使用实际实施架构的微小修改。将注意力算子应用于多维域上定义的函数的高昂成本导致需要更高效的注意力架构。为此,我们还引入了从计算机视觉中借鉴的补丁策略的函数空间泛化,并引入了一类相关的神经算子。在一系列算子学习问题上的数值结果展示了我们方法在注意力函数空间 formulation 中的潜力及其在神经算子中的应用。

英文摘要:

Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning. Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time series problems. Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators. Motivated by this, we study transformers in the function space setting. We formulate attention as a map between infinite dimensional function spaces and prove that the attention mechanism as implemented in practice is a Monte Carlo or finite difference approximation of this operator. The function space formulation allows for the design of transformer neural operators, a class of architectures designed to learn mappings between function spaces. In this paper, we state and prove the first universal approximation result for transformer neural operators, using only a slight modification of the architecture implemented in practice. The prohibitive cost of applying the attention operator to functions defined on multi-dimensional domains leads to the need for more efficient attention-based architectures. For this reason we also introduce a function space generalization of the patching strategy from computer vision, and introduce a class of associated neural operators. Numerical results, on an array of operator learning problems, demonstrate the promise of our approaches to function space formulations of attention and their use in neural operators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑