arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.18523cs.LG
收藏

DTFormer:一种基于Transformer的离散时间动态图表示学习方法

DTFormer: A Transformer-Based Method for Discrete-Time Dynamic Graph Representation Learning

Fudan University(复旦大学);University of California, Davis(加利福尼亚大学戴维斯分校)

Xi Chen, Yun Xiong, Siwei Zhang, Jiawei Zhang, Yao Zhang, Shiyang Zhou, Xixi Wu, Mingyang Zhang, Tengfei Liu, Weiqiang Wang

AI总结:

本文提出基于Transformer架构的DTFormer方法,利用注意力机制同时处理图拓扑与时间动态,并引入节点交集关系和多分块模块,在六个动态图基准上实现了SOTA性能。

摘要:

离散时间动态图(DTDGs)在现实世界的实现中非常普遍,并因其数据获取的便利性而备受学术界研究人员和工业界从业者的关注。DTDGs的表示学习被广泛应用于建模时间变化实体及其不断演变的连接的动态特性。目前,DTDG表示学习主要依赖于GNN+RNN架构,这暴露了图神经网络(GNNs)和循环神经网络(RNNs)固有的局限性。随着模型架构加深,GNNs会出现过平滑问题,而RNNs难以有效捕捉长期依赖关系。GNN+RNN架构在扩展到大型图尺寸和长序列时也面临困难。此外,这些方法通常分别计算节点表示,仅关注单个节点特征,从而忽略了待预测连边的两个节点之间的行为交集,例如这两个节点在同一上下文中同时出现或共享共同邻居的情况。本文提出了一种用于DTDGs的新型表示学习方法DTFormer,从传统的GNN+RNN框架转向基于Transformer的架构。我们的方法利用注意力机制并发处理每个时间戳下图的拓扑信息以及跨时间戳的图时间动态,规避了GNNs和RNNs上述的根本缺陷。此外,我们通过引入节点间的交集关系并集成多分块模块,增强了模型的表达能力。在六个公开动态图基准数据集上进行的大量实验证实了我们模型的有效性,实现了SOTA性能。

英文摘要:

Discrete-Time Dynamic Graphs (DTDGs), which are prevalent in real-world implementations and notable for their ease of data acquisition, have garnered considerable attention from both academic researchers and industry practitioners. The representation learning of DTDGs has been extensively applied to model the dynamics of temporally changing entities and their evolving connections. Currently, DTDG representation learning predominantly relies on GNN+RNN architectures, which manifest the inherent limitations of both Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs). GNNs suffer from the over-smoothing issue as the models architecture goes deeper, while RNNs struggle to capture long-term dependencies effectively. GNN+RNN architectures also grapple with scaling to large graph sizes and long sequences. Additionally, these methods often compute node representations separately and focus solely on individual node characteristics, thereby overlooking the behavior intersections between the two nodes whose link is being predicted, such as instances where the two nodes appear together in the same context or share common neighbors. This paper introduces a novel representation learning method DTFormer for DTDGs, pivoting from the traditional GNN+RNN framework to a Transformer-based architecture. Our approach exploits the attention mechanism to concurrently process topological information within the graph at each timestamp and temporal dynamics of graphs along the timestamps, circumventing the aforementioned fundamental weakness of both GNNs and RNNs. Moreover, we enhance the model's expressive capability by incorporating the intersection relationships among nodes and integrating a multi-patching module. Extensive experiments conducted on six public dynamic graph benchmark datasets confirm our model's efficacy, achieving the SOTA performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑