arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.01233cs.LG
收藏

GEN:面向长程图建模的图Transformer实用替代方案

GEN: A Practical Alternative to Graph Transformers for Long-Range Graph Modeling

School of Computer Science & Cyberspace Science, Xiangtan University(计算机科学与网络空间科学学院,湘潭大学);Hunan University(湖南大学)

Shuo Wang, Ge Cheng, Yun Zhang

AI总结:

针对MPNN长程信息传播弱、图Transformer扩展性差的问题,提出结合边级与跳级自注意力、基于图消去算法避免重复计数的GEN,在长程图基准上性能优异且支持大图全批量训练。

摘要:

消息传递神经网络(MPNNs)能有效建模局部关系,但难以在长距离上传播信息。图Transformer(GTs)通过全局自注意力缓解了这一问题,但其计算成本随节点数呈二次增长,限制了可扩展性。我们提出图消去网络(Graph Elimination Networks, GENs),这是一种MPNN变体,可近似实现类GT的长程建模能力,同时保持高效率。GENs并行结合边级和跳级自注意力;二者的乘法组合生成了在有界K跳感受野内、可沿边因子和跳因子分解的注意力核。为实现跳级注意力,我们提出图消去算法(Graph Elimination Algorithm, GEA),该算法可避免跨跳的重复计数,确保每一轮仅注入一次k跳增量贡献。通过计算连续轮次间的差值,可得到k跳增量,并生成解耦的多跳特征作为跳级注意力的输入。这保留了不同跳距离间更清晰的结构差异,能更忠实地建模K跳邻域内远距离节点间的成对依赖关系。在长程图基准(Long-Range Graph Benchmark, LRGB)上,GENs在PascalVOC-SP和COCO-SP任务上分别比强MPNN基线高出7.7和6.0个百分点(pp),且性能与最先进的图Transformer相当或更优。在OGBN-Products数据集上,GENs支持全批量训练/推理,而Exphormer等稀疏注意力基线在相近预算下受内存限制,凸显了GENs作为大型稀疏图实用替代方案的价值。

英文摘要:

Message Passing Neural Networks (MPNNs) model local relations effectively but struggle to propagate information over long distances. Graph Transformers (GTs) mitigate this via global self-attention, yet their quadratic cost in the number of nodes limits scalability. We propose Graph Elimination Networks (GENs), an MPNN variant that approximates GT-like long-range modeling while maintaining high efficiency. GENs combine edge-wise and hop-wise self-attention in parallel; their multiplicative composition yields an attention kernel separable across edge and hop factors within a bounded K-hop receptive field. To enable hop-wise attention, we introduce the Graph Elimination Algorithm (GEA), which prevents double counting across hops, ensuring that each round injects the k-hop incremental contribution exactly once. Taking differences between successive rounds recovers the k-hop increment and yields disentangled multi-hop features as inputs for hop-wise attention. This preserves clearer structural distinctions across hop distances and enables more faithful modeling of pairwise dependencies between distant nodes within the K-hop neighborhood. On the Long-Range Graph Benchmark (LRGB), GENs outperform strong MPNN baselines by 7.7 and 6.0 percentage points (pp) on PascalVOC-SP and COCO-SP, and achieve performance on par with or better than state-of-the-art Graph Transformers. On OGBN-Products, GENs support full-batch training/inference, while sparse-attention baselines like Exphormer struggle with memory limits under comparable budgets, highlighting GENs as a practical alternative for large, sparse graphs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑