Many graph representation learning (GRL) problems are dynamic, with millions of edges added or removed per second. A fundamental workload in this setting is dynamic link prediction: using a history of graph updates to predict whether a given pair of vertices will become connected. Recent schemes for link prediction in such dynamic settings employ Transformers, modeling individual graph updates as single tokens. In this work, we propose HOT: a model that enhances this line of works by harnessing higher-order (HO) graph structures; specifically, k-hop neighbors and more general subgraphs containing a given pair of vertices. Harnessing such HO structures by encoding them into the attention matrix of the underlying Transformer results in higher accuracy of link prediction outcomes, but at the expense of increased memory pressure. To alleviate this, we resort to a recent class of schemes that impose hierarchy on the attention matrix, significantly reducing memory footprint. The final design offers a sweetspot between high accuracy and low memory utilization. HOT outperforms other dynamic GRL schemes, for example achieving 9%, 7%, and 15% higher accuracy than - respectively - DyGFormer, TGN, and GraphMixer, for the MOOC dataset. Our design can be seamlessly extended towards other dynamic GRL workloads.
Higher-Order Transformers With Kronecker-Structured Attention
Mila McGill University(Mila 麦吉尔大学);Mila, CIFAR AI Chair McGill University(Mila,CIFAR AI 职位 麦吉尔大学);Mila - DIRO, CIFAR AI Chair University of Montreal(Mila,DIRO,CIFAR AI 职位 蒙特利尔大学)
Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation
National Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School(北京大学深圳研究生院通用人工智能国家重点实验室);Amazon Prime Video(亚马逊Prime视频);University of Trento(特伦托大学)
HOTCAKE: Higher Order Tucker Articulated Kernels for Deeper CNN Compression
Institute for Clarity in Documentation(清晰度文献研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)