We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks. The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process. This modularity enhances parameter correspondence control, making HAMLET adaptable to PDEs of arbitrary geometries and varied input formats. Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness. HAMLET is not just tailored to a single type of physical simulation, but can be applied across various domains. Moreover, it boosts model resilience and performance, especially in scenarios with limited data. We demonstrate, through extensive experiments, that our framework is capable of outperforming current techniques for PDEs.
NTFormer: A Composite Node Tokenized Graph Transformer for Node Classification
School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);Institute of Artificial Intelligence, Huazhong University of Science and Technology(华中科技大学人工智能研究院);Hopcroft Center on Computing Science, Huazhong University of Science and Technology(华中科技大学霍普克罗夫特计算科学中心)
在本文中,我们首次将Attention Is All You Need中最先进的Transformer中的自注意力应用于与偏微分方程相关的数据驱动算子学习问题。努力放在一起解释的原理,并提高注意力机制的功效。利用Hilbert空间中的算子逼近理论,首次证明了在标度点积注意力中的softmax归一化是充分的,但不是必要的。在没有softmax的情况下,可以证明线性化的Transformer变体的近似能力与逐层的Petrov-Galerkin投影相当,并且估计与序列长度无关。提出了一种新的层归一化方案,该方案模仿Petrov-Galerkin投影,以允许缩放通过注意层传播,这有助于模型在具有未归一化数据的算子学习任务中实现显着的准确性。最后,我们提出了三个操作学习实验,包括粘性Burgers方程,界面达西流,和反界面系数识别问题。新提出的简单的基于注意力的算子学习器Galerkin Transformer,在训练成本和评估精度方面都比其softmax归一化的同类算法有了显着的改进。
Training-Free Transformer Merging via Sequential Local Operator Alignment
CISPA Helmholtz Center for Information Security(CISPA赫尔姆霍茨信息安全中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Massachusetts Institute of Technology(麻省理工学院)
ShapeFormer: Transformer-based Shape Completion via Sparse Representation
Shenzhen University(深圳大学);University College London(伦敦大学学院);Adobe Research(奥多比研究院);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Tel Aviv University(特拉维夫大学)
Physics and Data Driven Transformer-Mamba Framework for Flow Field
National University of Defense Technology(国防科技大学);Nanjing Agricultural University(南京农业大学);National Key Laboratory of Parallel and Distributed Computing(并行与分布式计算国家重点实验室);National SuperComputer Center in Tianjin(国家超级计算天津中心)