arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.12487cs.LG
收藏

通过正交注意力改进算子学习

Improved Operator Learning by Orthogonal Attention

Qing Yuan Research Institute(清源研究院);SEIEE, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);Dept. of Comp. Sci. & Tech., Tsinghua University(清华大学计算机科学与技术系)

Zipeng Xiao, Zhongkai Hao, Bokai Lin, Zhijie Deng, Hang Su

AI总结:

本文提出基于核积分算子特征分解的正交注意力机制,通过正则化缓解注意力神经算子的过拟合问题,并在六个标准基准数据集上验证了其优于现有方法的泛化性能。

摘要:

神经算子作为学习偏微分方程解的高效代理模型,在科学机器学习领域受到了广泛关注。其中,基于注意力的神经算子已成为相关研究的主流之一。然而,现有方法由于注意力机制中参数数量庞大,容易对有限的训练数据产生过拟合。为了解决这一问题,我们基于核积分算子的特征分解和特征函数的神经逼近,开发了一种正交注意力机制。正交化自然地对所得神经算子施加了适当的正则化效果,有助于抵抗过拟合并提升泛化能力。在包含规则和不规则几何形状的六个标准神经算子基准数据集上的实验表明,我们的方法能够以可观的幅度超越竞争基线。

英文摘要:

Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑