arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.18526cs.LGcs.SI
收藏

HOT:借助高效 Transformer 的高阶动态图表示学习

HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformers

ETH Zurich(苏黎世联邦理工学院);Cledar(克莱达(公司名))

Maciej Besta, Afonso Claudino Catarino, Lukas Gianinazzi, Nils Blach, Piotr Nyczyk, Hubert Niewiadomski, Torsten Hoefler

AI总结:

本文提出 HOT,通过在 Transformer 注意力中利用 k 跳邻居和子图等高阶结构,并结合层次化注意力降低内存,实现更准确高效的动态链路预测。

摘要:

许多图表示学习(GRL)问题都是动态的,每秒有数百万条边被添加或删除。这种场景下的一项基础工作负载是动态链路预测:利用图更新的历史记录来预测给定的一对顶点是否会相连。近期针对此类动态场景中链路预测的方案采用了 Transformers,将单个图更新建模为单一 token。在本工作中,我们提出了 HOT:一个通过利用高阶(HO)图结构来增强这一系列工作的模型;具体而言,这些结构包括 k 跳邻居以及包含给定顶点对的更一般子图。通过将此类 HO 结构编码到底层 Transformer 的注意力矩阵中来利用它们,可以提高链路预测结果的准确率,但代价是内存压力增加。为缓解这一问题,我们采用了近期一类在注意力矩阵上施加层次结构的方案,从而显著降低内存占用。最终设计在高准确率与低内存利用率之间提供了一个理想平衡点。HOT 优于其他动态 GRL 方案,例如在 MOOC 数据集上,其准确率分别比 DyGFormer、TGN 和 GraphMixer 高出 9%、7% 和 15%。我们的设计可以无缝扩展到其他动态 GRL 工作负载。

英文摘要:

Many graph representation learning (GRL) problems are dynamic, with millions of edges added or removed per second. A fundamental workload in this setting is dynamic link prediction: using a history of graph updates to predict whether a given pair of vertices will become connected. Recent schemes for link prediction in such dynamic settings employ Transformers, modeling individual graph updates as single tokens. In this work, we propose HOT: a model that enhances this line of works by harnessing higher-order (HO) graph structures; specifically, k-hop neighbors and more general subgraphs containing a given pair of vertices. Harnessing such HO structures by encoding them into the attention matrix of the underlying Transformer results in higher accuracy of link prediction outcomes, but at the expense of increased memory pressure. To alleviate this, we resort to a recent class of schemes that impose hierarchy on the attention matrix, significantly reducing memory footprint. The final design offers a sweetspot between high accuracy and low memory utilization. HOT outperforms other dynamic GRL schemes, for example achieving 9%, 7%, and 15% higher accuracy than - respectively - DyGFormer, TGN, and GraphMixer, for the MOOC dataset. Our design can be seamlessly extended towards other dynamic GRL workloads.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑