Discrete-Time Dynamic Graphs (DTDGs), which are prevalent in real-world implementations and notable for their ease of data acquisition, have garnered considerable attention from both academic researchers and industry practitioners. The representation learning of DTDGs has been extensively applied to model the dynamics of temporally changing entities and their evolving connections. Currently, DTDG representation learning predominantly relies on GNN+RNN architectures, which manifest the inherent limitations of both Graph Neural Networks (GNNs) and Recurrent Neural Networks (RNNs). GNNs suffer from the over-smoothing issue as the models architecture goes deeper, while RNNs struggle to capture long-term dependencies effectively. GNN+RNN architectures also grapple with scaling to large graph sizes and long sequences. Additionally, these methods often compute node representations separately and focus solely on individual node characteristics, thereby overlooking the behavior intersections between the two nodes whose link is being predicted, such as instances where the two nodes appear together in the same context or share common neighbors.
This paper introduces a novel representation learning method DTFormer for DTDGs, pivoting from the traditional GNN+RNN framework to a Transformer-based architecture. Our approach exploits the attention mechanism to concurrently process topological information within the graph at each timestamp and temporal dynamics of graphs along the timestamps, circumventing the aforementioned fundamental weakness of both GNNs and RNNs. Moreover, we enhance the model's expressive capability by incorporating the intersection relationships among nodes and integrating a multi-patching module. Extensive experiments conducted on six public dynamic graph benchmark datasets confirm our model's efficacy, achieving the SOTA performance.
MSDformer: Multi-scale Discrete Transformer For Time Series Generation
Nanyang Technological University(南洋理工大学);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tencent(腾讯);Peng Cheng Laboratory(鹏城实验室);Institute of Microelectronics, Chinese Academy of Sciences(中国科学院微电子研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院)
HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting
College of Computing and Data Science, Nanyang Technological University (NTU)(南洋理工大学计算与数据科学学院);Webank-NTU Joint Research Institute on Fintech, NTU(微众银行-南洋理工大学金融科技联合研究院);Tencent AI Lab(腾讯人工智能实验室)
Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers
Machine and Hybrid Intelligence Lab, Northwestern University(机器与混合智能实验室,西北大学);Department of Electrical and Computer Engineering, University of Illinois Chicago(电气与计算机工程系,伊利诺伊大学香槟分校)
Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting
Physics Transformer: Tailoring Transformer for General PDE Prediction
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Mechanics and Engineering Science, Peking University(北京大学力学与工程科学学院)
OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Oden Institute for Computational Engineering and Sciences, University of Texas, Austin(德克萨斯大学奥斯汀分校奥登计算工程与科学学院)
ShapeFormer: Transformer-based Shape Completion via Sparse Representation
Shenzhen University(深圳大学);University College London(伦敦大学学院);Adobe Research(奥多比研究院);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Tel Aviv University(特拉维夫大学)
HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
Mila, Quebec AI Institute(Mila-魁北克人工智能研究所);Concordia University(康考迪亚大学);Sapienza University of Rome(罗马大学);Inria, Université Grenoble Alpes CNRS, LJK(法国国家信息与自动化研究所,格勒诺布尔阿尔卑斯大学国家科学研究中心,让·库尔曼实验室)
Diagnostic Spatio-temporal Transformer with Faithful Encoding
Ikerlan Technology Research Centre(伊克兰技术研究中心);Basque Research and Technology Alliance (BRTA)(巴斯克研究与技术联盟(BRTA));Mondragon University(蒙德拉贡大学);IBM Research(IBM研究院)
Dayforce Supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), through grant DGECR-2022-04531. H.C.M.(Dayforce);Department of Mathematics and Statistics York University(约克大学数学与统计学系);Serrano Academy(塞拉诺学院)
DASViT: Differentiable Architecture Search for Vision Transformer
School of Computer Science and Electronic Engineering University of Surrey(计算机科学与电子工程学院大学 Surrey 大学);School of Artificial Intelligence and Computer Science Jiangnan University(人工智能与计算机科学学院江南大学)