arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.17569stat.MLcs.LG
收藏

高斯过程极限揭示图变换器的结构优势

Gaussian Process Limit Reveals Structural Benefits of Graph Transformers

Nil Ayday, Lingchu Yang, Debarghya Ghoshdastidar

AI总结:

本文通过高斯过程极限分析,揭示图变换器在节点级预测任务中优于图卷积网络的结构优势,证明其能保留社区信息并防止过平滑。

摘要:

图变换器是处理图结构数据的最新方法,已知能避免消息传递架构的多个问题,但其理论依据有限。本文证明,在节点级预测任务中,基于注意力的架构在结构上优于图卷积网络。具体而言,我们研究了无限宽度和无限头数的图变换器(GAT、Graphormer、Specformer)的神经网络高斯过程极限,并推导出节点级和边级的核函数。我们的结果描述了节点特征和图结构如何通过图注意力层传播。作为具体例子,我们证明图变换器在结构上能保留社区信息,并在深层中维持判别性节点表示,从而防止过平滑。我们在合成和现实图上提供了实证证据,验证了我们的理论见解,例如整合信息性先验和位置编码可以提高深度图变换器的性能。

英文摘要:

Graph transformers are the state-of-the-art for learning from graph-structured data and are empirically known to avoid several pitfalls of message-passing architectures. However, there is limited theoretical analysis on why these models perform well in practice. In this work, we prove that attention-based architectures have structural benefits over graph convolutional networks in the context of node-level prediction tasks. Specifically, we study the neural network gaussian process limits of graph transformers (GAT, Graphormer, Specformer) with infinite width and infinite heads, and derive the node-level and edge-level kernels across the layers. Our results characterise how the node features and the graph structure propagate through the graph attention layers. As a specific example, we prove that graph transformers structurally preserve community information and maintain discriminative node representations even in deep layers, thereby preventing oversmoothing. We provide empirical evidence on synthetic and real-world graphs that validate our theoretical insights, such as integrating informative priors and positional encoding can improve performance of deep graph transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑