arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.03582cs.LG
收藏

简单而有效:神经算子中的低秩空间注意力

Simple yet Effective: Low-Rank Spatial Attention for Neural Operators

University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院)

Zherui Yang, Haiyang Xin, Tao Du, Ligang Liu

AI总结:

本文提出低秩空间注意力模块,通过压缩高维特征到紧凑潜在空间,实现高效全局交互,提升神经算子在求解偏微分方程中的准确性与效率。

摘要:

神经算子已作为数据驱动的替代方案,用于求解偏微分方程,其成功依赖于有效建模由底层物理引起的长程全局耦合。在许多PDE领域中,诱导的全局交互核经验上是可压缩的,表现出快速谱衰减,允许低秩近似。我们利用这一观察,将神经算子中的代表性全局混合模块统一到一个共享的低秩模板下:将高维点特征压缩到紧凑的潜在空间,在此空间中处理全局交互,并将全局上下文重建回空间点。基于此观点,我们引入低秩空间注意力(LRSA)作为该模板的简洁直接实例。关键的是,与以往依赖非标准聚合或归一化模块的方法不同,LRSA纯粹由标准Transformer原始组件构建,即注意力、归一化和前馈网络,产生一个简洁的块,易于实现且直接与硬件优化内核兼容。在我们的实验中,这种简单的构造足以实现高精度,相对于第二好的方法,平均误差减少了超过17%,同时在混合精度训练中保持稳定和高效。

英文摘要:

Neural operators have emerged as data-driven surrogates for solving partial differential equations (PDEs), and their success hinges on efficiently modeling the long-range, global coupling among spatial points induced by the underlying physics. In many PDE regimes, the induced global interaction kernels are empirically compressible, exhibiting rapid spectral decay that admits low-rank approximations. We leverage this observation to unify representative global mixing modules in neural operators under a shared low-rank template: compressing high-dimensional pointwise features into a compact latent space, processing global interactions within it, and reconstructing the global context back to spatial points. Guided by this view, we introduce Low-Rank Spatial Attention (LRSA) as a clean and direct instantiation of this template. Crucially, unlike prior approaches that often rely on non-standard aggregation or normalization modules, LRSA is built purely from standard Transformer primitives, i.e., attention, normalization, and feed-forward networks, yielding a concise block that is straightforward to implement and directly compatible with hardware-optimized kernels. In our experiments, such a simple construction is sufficient to achieve high accuracy, yielding an average error reduction of over 17\% relative to second-best methods, while remaining stable and efficient in mixed-precision training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑