arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09603cs.LGcs.AI
收藏

可扩展的图自监督学习

Scalable Graph Self-Supervised Learning

University of Waterloo(滑铁卢大学);University of Toronto(多伦多大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Ali Saheb Pasand, Reza Moravej, Mahdi Biparva, Raika Karimi, Ali Ghodsi

AI总结:

针对图自监督学习中损失计算复杂度随节点数和维度增加的问题,提出基于节点或维度采样的方法降低协方差矩阵计算成本,实验证明该方法在保持甚至提升下游性能的同时有效减少了计算开销。

摘要:

在图的正则化自监督学习(SSL)方法中,计算复杂度随着图中节点数量和嵌入维度的增加而上升。为了缓解非对比图SSL的可扩展性问题,我们提出了一种新方法,以降低带有体积最大化项的预训练损失函数中计算协方差矩阵的成本。我们的工作重点是通过图节点或维度采样来降低与损失计算相关的成本。我们提供了理论见解,说明为何维度采样能够实现准确的损失计算,并通过新方法的数学推导对其进行了支持。我们在节点级图预测任务上开发了实验设置,由于现实世界图的庞大尺寸,SSL预训练在这些任务中被证明具有挑战性。我们的实验表明,通过节点或维度采样可以降低与损失计算相关的成本,而不会降低下游性能。结果表明,采样在大多数情况下能提升下游性能。我们还提供了消融研究和实验分析,以理清实验设置中不同因素的作用。

英文摘要:

In regularization Self-Supervised Learning (SSL) methods for graphs, computational complexity increases with the number of nodes in graphs and embedding dimensions. To mitigate the scalability of non-contrastive graph SSL, we propose a novel approach to reduce the cost of computing the covariance matrix for the pre-training loss function with volume-maximization terms. Our work focuses on reducing the cost associated with the loss computation via graph node or dimension sampling. We provide theoretical insight into why dimension sampling would result in accurate loss computations and support it with mathematical derivation of the novel approach. We develop our experimental setup on the node-level graph prediction tasks, where SSL pre-training has shown to be difficult due to the large size of real world graphs. Our experiments demonstrate that the cost associated with the loss computation can be reduced via node or dimension sampling without lowering the downstream performance. Our results demonstrate that sampling mostly results in improved downstream performance. Ablation studies and experimental analysis are provided to untangle the role of the different factors in the experimental setup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑