Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins.
Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport
Georgia Institute of Technology(佐治亚理工学院)
Stiefel流形上的优化问题,即,满足正交性约束的(不一定是正方形的)矩阵的最小化函数已经被广泛研究。然而,提出了一种新的方法的基础上,第一次,精心设计的连续和离散动态之间的相互作用。它导致了一个基于梯度的优化器,具有内在的附加动量。该方法精确地保持了流形结构,但不需要额外的操作来保持变化的(余)切空间中的动量,因此具有较低的计算成本和令人满意的精度。它的推广自适应学习率也证明。在实际任务中观察到显著的性能。例如,我们发现,当使用我们的优化器时,对从头开始训练的视觉Transformer [Dosovitskiy et al. 2022]的注意力头部设置正交约束可以显著提高其性能,并且每个头部最好在其自身内正交,但不一定与其他头部正交。该优化器还使投影鲁棒Wasserstein距离[Paty & Cuturi 2019; Lin et al. 2020]的有用概念对于高维度最优传输更加有效。
An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction
Zhejiang University(浙江大学);Tianjin Jinhang Technical Physics Institute(天津津航技术物理研究所);Complex System Control and Intelligent Agent Cooperative Laboratory(复杂系统控制与智能体协同实验室)
Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
Tencent AI Lab(腾讯AI实验室);Megvii Research Shanghai(上海旷视研究院);Tsinghua University(清华大学);Peking University(北京大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification
Australian Institute for Machine Learning(澳大利亚机器学习研究所);University of Adelaide(阿德莱德大学);Peter Faber Business School, Australian Catholic University(澳大利亚天主教大学彼得·法伯商学院);Australian Catholic University(澳大利亚天主教大学);Amsterdam UMC(阿姆斯特丹大学医学中心);Flinders Medical Centre(弗林德斯医学中心);Flinders University(弗林德斯大学);Groningen University Medical Centre(格罗宁根大学医学中心)
Quantum Attention for Vision Transformers in High Energy Physics
University of Pavia(帕维亚大学);Indian Institute of Technology Bhilai(贝利特印度理工学院);University of Alabama(阿拉巴马大学);University of Kansas(堪萨斯大学);Technische Universität München(慕尼黑工业大学);University of Florida(佛罗里达大学)