arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.01102cs.LGcs.AI
收藏

少即是多:图Transformer中的过度全局化问题

Less is More: on the Over-Globalizing Problem in Graph Transformers

School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);School of Software, Beihang University(北京航空航天大学软件学院)

Yujie Xing, Xiao Wang, Yibo Li, Hai Huang, Chuan Shi

AI总结:

本文揭示图Transformer的过度全局化问题,提出双层全局图Transformer与协作训练(CoBFormer),兼顾远近节点信息提取,实验验证其有效性。

摘要:

图Transformer凭借其全局注意力机制,已成为处理图结构数据的新工具。众所周知,全局注意力机制在全连接图中考虑了更广泛的感受野,这使许多人相信可以从所有节点中提取有用信息。在本文中,我们质疑这一观点:全局化特性是否总是有利于图Transformer?我们通过实证证据和理论分析揭示了图Transformer中的过度全局化问题,即当前的注意力机制过度关注那些远距离节点,而实际上包含大部分有用信息的邻近节点则相对被弱化。然后,我们提出了一种新颖的双层全局图Transformer与协作训练(CoBFormer),包括簇间和簇内Transformer,以防止过度全局化问题,同时保持从远距离节点提取有价值信息的能力。此外,我们提出了协作训练,以在理论保证下提高模型的泛化能力。在各种图上的大量实验充分验证了我们提出的CoBFormer的有效性。

英文摘要:

Graph Transformer, due to its global attention mechanism, has emerged as a new tool in dealing with graph-structured data. It is well recognized that the global attention mechanism considers a wider receptive field in a fully connected graph, leading many to believe that useful information can be extracted from all the nodes. In this paper, we challenge this belief: does the globalizing property always benefit Graph Transformers? We reveal the over-globalizing problem in Graph Transformer by presenting both empirical evidence and theoretical analysis, i.e., the current attention mechanism overly focuses on those distant nodes, while the near nodes, which actually contain most of the useful information, are relatively weakened. Then we propose a novel Bi-Level Global Graph Transformer with Collaborative Training (CoBFormer), including the inter-cluster and intra-cluster Transformers, to prevent the over-globalizing problem while keeping the ability to extract valuable information from distant nodes. Moreover, the collaborative training is proposed to improve the model's generalization ability with a theoretical guarantee. Extensive experiments on various graphs well validate the effectiveness of our proposed CoBFormer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑