arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19622cs.LGcs.AI
收藏

VecFormer: 向高效且通用的图变换器迈进:基于图标记注意力的图变换器

VecFormer: Towards Efficient and Generalizable Graph Transformer with Graph Token Attention

Zhejiang University(浙江大学);Westlake University(西湖大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Tongji University(同济大学);ShanghaiTech University(上海科技大学);Southern University of Science and Technology(南方科技大学);Jilin University(吉林大学)

Jingbo Zhou, Jun Xia, Siyuan Li, Yunfan Liu, Wenjun Wang, Yufei Huang, Changxi Chi, Mutian Hong, Zhuoli Ouyang, Shu Wang, Zhongqi Wang, Xingyu Wu, Chang Yu, Stan Z. Li

AI总结:

VecFormer通过两阶段训练和图标记注意力机制,提升图变换器在大规模图和分布外场景中的效率与泛化能力。

摘要:

图变换器在图表示学习领域展现出了令人印象深刻的能力。然而,现有方法面临两个关键挑战:(1)大多数模型存在计算复杂度呈指数级增长的问题,使得难以扩展到大规模图;(2)基于节点级操作的注意力机制限制了模型的灵活性,并在分布外(OOD)场景中导致泛化性能较差。为了解决这些问题,我们提出了VecFormer(Vector Quantized Graph TransFormer),一种高效且高度通用的节点分类模型,特别是在分布外设置中。VecFormer采用两阶段训练范式。在第一阶段,使用两个代码本来重建节点特征和图结构,旨在学习丰富的语义Graph Codes。在第二阶段,基于变换后的交叉代码本在Graph Token层面执行注意力机制,从而降低计算复杂度并增强模型的泛化能力。在各种规模的数据集上进行的广泛实验表明,VecFormer在性能和速度上均优于现有的图变换器。

英文摘要:

Graph Transformer has demonstrated impressive capabilities in the field of graph representation learning. However, existing approaches face two critical challenges: (1) most models suffer from exponentially increasing computational complexity, making it difficult to scale to large graphs; (2) attention mechanisms based on node-level operations limit the flexibility of the model and result in poor generalization performance in out-of-distribution (OOD) scenarios. To address these issues, we propose \textbf{VecFormer} (the \textbf{Vec}tor Quantized Graph Trans\textbf{former}), an efficient and highly generalizable model for node classification, particularly under OOD settings. VecFormer adopts a two-stage training paradigm. In the first stage, two codebooks are used to reconstruct the node features and the graph structure, aiming to learn the rich semantic \texttt{Graph Codes}. In the second stage, attention mechanisms are performed at the \texttt{Graph Token} level based on the transformed cross codebook, reducing computational complexity while enhancing the model's generalization capability. Extensive experiments on datasets of various sizes demonstrate that VecFormer outperforms the existing Graph Transformer in both performance and speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑