DHIL-GT: 基于解耦层次标记的可扩展图Transformer
DHIL-GT: Scalable Graph Transformer with Decoupled Hierarchy Labeling
AI总结:
针对图Transformer的图规模计算瓶颈,提出DHIL-GT:通过解耦图计算至独立预计算阶段,利用图标记层次提取信息,实现线性复杂度,在大规模基准上计算效率更优,且在同/异质性图上效果顶级。
摘要:
图Transformer(GT)是近年来用于学习图结构数据的一种极具潜力的神经网络架构。然而,其全局注意力机制随图规模呈二次复杂度增长,限制了它在大型图上的广泛应用。现有方法尝试通过改变模型架构或编码层次化图数据来提升GT的可扩展性,但我们的分析表明,这些模型仍受限于与图规模相关的计算瓶颈。在本研究中,我们针对GT的可扩展性问题,提出了DHIL-GT——一种可扩展的图Transformer,它通过将图计算完全解耦并提前置于独立阶段来简化网络学习。DHIL-GT利用图标记技术有效提取层次信息,我们证明图标记层次比普通邻接矩阵更具信息性:它既提供全局连接,又增强局部性,尤其适用于处理异质性等复杂图模式。我们进一步设计了子图采样和位置编码方案,基于图标记以端到端方式预计算模型输入。因此,训练阶段无需进行图相关计算,实现了理想的小批量处理能力和GPU利用率。值得注意的是,DHIL-GT的预计算和训练过程复杂度分别与图的边数和节点数呈线性关系。大量实验表明,在大规模基准测试中,DHIL-GT在计算效率和小批量处理能力上优于现有可扩展图Transformer设计,同时在同质性和异质性图上均达到顶级效果。
英文摘要:
Graph Transformer (GT) has recently emerged as a promising neural network architecture for learning graph-structured data. However, its global attention mechanism with quadratic complexity concerning the graph scale prevents wider application to large graphs. While current methods attempt to enhance GT scalability by altering model architecture or encoding hierarchical graph data, our analysis reveals that these models still suffer from the computational bottleneck related to graph-scale operations. In this work, we target the GT scalability issue and propose DHIL-GT, a scalable Graph Transformer that simplifies network learning by fully decoupling the graph computation to a separate stage in advance. DHIL-GT effectively retrieves hierarchical information by exploiting the graph labeling technique, as we show that the graph label hierarchy is more informative than plain adjacency by offering global connections while promoting locality, and is particularly suitable for handling complex graph patterns such as heterophily. We further design subgraph sampling and positional encoding schemes for precomputing model input on top of graph labels in an end-to-end manner. The training stage thus favorably removes graph-related computations, leading to ideal mini-batch capability and GPU utilization. Notably, the precomputation and training processes of DHIL-GT achieve complexities linear to the number of graph edges and nodes, respectively. Extensive experiments demonstrate that DHIL-GT is efficient in terms of computational boost and mini-batch capability over existing scalable Graph Transformer designs on large-scale benchmarks, while achieving top-tier effectiveness on both homophilous and heterophilous graphs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。