基于区域分解的层次注意力机制
Overlapping Schwarz Attention: Hierarchical Attention via Domain Decomposition
AI总结：
提出一种基于两水平重叠Schwarz区域分解的层次注意力机制，通过局部低秩注意力块与粗网格注意力块结合，在少参数下实现更快训练和更高精度。
摘要：
我们提出了一种基于两水平重叠Schwarz区域分解的层次注意力机制。该方法的动机源于观察到两水平Schwarz区域分解方法将局部子域校正与一个传达全局、长程信息的粗水平相结合。我们在一个具有齐次Dirichlet边界条件的一维扩散问题背景下，测试了其在有限维算子学习中的实用性。尽管该问题简单，但它提供了一个受控的序列到序列设置，其中精确的非局部解算子已知。离散化后，学习解算子相当于逼近一个对称正定矩阵的逆。作为基线，我们使用一个全局无softmax的低秩注意力算子，形式为$QK^T$。所提出的构造将这个密集的全局分解替换为一个两水平加性结构：重叠子域上的局部低秩注意力块与一个粗注意力块相结合。得到的算子形式为$$M_{\theta}^{-1} = \Phi Q_0 K_0^T \Phi^T + \sum_{i=1}^{N} R_i^T D_i^{1/2} Q_i K_i^T D_i^{1/2} R_i.$$ 这里$R_i$限制到重叠子域，$D_i$是单位划分权重，$\Phi$是粗插值（或延拓）矩阵。针对合成Fourier右端项的数值实验表明，区域分解注意力算子能够比全局低秩注意力基线训练更快，并在使用显著更少参数的情况下提供更精确的逼近。
英文摘要：
We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition. The method is motivated by domain decomposition methods in partial differential equations which combine local subdomain corrections with a coarse level that communicates global, long-range information. We test its usefulness in the context of finite-dimensional operator learning using a simple, one-dimensional diffusion problem. Although elementary, this problem provides a controlled sequence-to-sequence setting in which the exact nonlocal solution operator is known. After discretization, learning the solution operator amounts to approximating the inverse of a symmetric positive definite matrix. As a baseline, we use a global softmax-free low-rank attention operator of the form \(QK^T\). The proposed construction replaces this global factorization by a two-level additive structure: local low-rank attention blocks on overlapping subdomains are combined with a coarse attention block. The resulting operator has the form $$ M_θ^{-1} = ΦQ_0 K_0^T Φ^T + \sum_{i=1}^{N} R_i^T D_i^{1/2} Q_i K_i^T D_i^{1/2} R_i . $$ Here, $R_i$ restricts to an overlapping subdomain, $D_i$ is a partition-of-unity weight, and $Φ$ is a coarse interpolation %(or prolongation) matrix. Numerical experiments for synthetic Fourier right-hand sides indicate that the domain-decomposition attention operator can converge faster and can give more accurate approximations than a global low-rank attention baseline while using significantly fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
