arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12059cs.LGcs.SI
收藏

图Transformer的一种可扩展且有效的替代方案

A Scalable and Effective Alternative to Graph Transformers

Georgia Institute of Technology(佐治亚理工学院);Meta AI;Amazon Web Services(亚马逊网络服务)

Kaan Sancak, Zhigang Hua, Jin Fang, Yan Xie, Andrey Malevich, Bo Long, Muhammed Fatih Balin, Ümit V. Çatalyürek

AI总结:

针对图Transformer二次复杂度难以适配大规模图的问题,提出可扩展的GECO方法,结合邻域传播与全局卷积准线性捕获长程依赖,在200万节点图上加速169倍,性能优于基线最高达4.5%。

摘要:

图神经网络(GNNs)在图表示学习中已展现出优异性能,但受限于表达能力,在捕获长程依赖方面面临挑战。为解决这一问题,研究者提出了图Transformer(GTs),利用自注意力机制有效建模节点间的成对关系。尽管具备优势,GTs的复杂度与图中节点数呈二次关系,阻碍了其在大规模图上的应用。本研究提出Graph-Enhanced Contextual Operator(GECO,图增强上下文算子),作为GTs的可扩展有效替代方案,它通过邻域传播与全局卷积,在准线性时间内有效捕获局部与全局依赖。我们在合成数据集上的研究表明,在含200万个节点的图上,GECO相较于优化后的注意力机制实现了169倍的加速。在多种基准上的进一步评估显示,GECO可扩展至传统GTs常面临内存与时间限制的大规模图。值得注意的是,GECO的性能始终与基线方法相当或更优,将当前最优水平(SOTA)最高提升了4.5%,为大规模图学习提供了一种可扩展且有效的解决方案。

英文摘要:

Graph Neural Networks (GNNs) have shown impressive performance in graph representation learning, but they face challenges in capturing long-range dependencies due to their limited expressive power. To address this, Graph Transformers (GTs) were introduced, utilizing self-attention mechanism to effectively model pairwise node relationships. Despite their advantages, GTs suffer from quadratic complexity w.r.t. the number of nodes in the graph, hindering their applicability to large graphs. In this work, we present Graph-Enhanced Contextual Operator (GECO), a scalable and effective alternative to GTs that leverages neighborhood propagation and global convolutions to effectively capture local and global dependencies in quasilinear time. Our study on synthetic datasets reveals that GECO reaches 169x speedup on a graph with 2M nodes w.r.t. optimized attention. Further evaluations on diverse range of benchmarks showcase that GECO scales to large graphs where traditional GTs often face memory and time limitations. Notably, GECO consistently achieves comparable or superior quality compared to baselines, improving the SOTA up to 4.5%, and offering a scalable and effective solution for large-scale graph learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑