arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.05337cs.LGcs.DC
收藏

DistDGL:分布式图神经网络训练十亿级图

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

AWS AI(AWS 人工智能部门);AWS Shanghai AI Lab(AWS 上海人工智能实验室)

Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, George Karypis

摘要:

图神经网络(GNN)在从图结构数据中学习方面取得了巨大的成功。它们被广泛用于各种应用,例如推荐、欺诈检测和搜索。在这些域中,图通常很大,包含数亿个节点和数十亿条边。为了应对这一挑战,我们开发了DistDGL,这是一个在机器集群上以小批量方式训练GNN的系统。DistDGL基于Deep Graph Library(DGL),这是一个流行的GNN开发框架。DistDGL将图及其相关数据(初始特征和嵌入)分布在机器上,并使用此分布通过遵循所有者计算规则来导出计算分解。DistDGL遵循同步训练方法,并允许形成小批量的自我网络包含非本地节点。为了最大限度地减少与分布式计算相关的管理费用,DistDGL使用高质量且轻量级的最小割图划分算法以及多个平衡约束。这允许它减少通信开销并静态平衡计算。它通过复制光环节点和使用稀疏嵌入更新进一步减少了通信。这些设计选择的组合使DistDGL能够训练高质量的模型,同时实现高并行效率和内存可扩展性。我们展示了我们的优化归纳和transductive GNN模型。我们的研究结果表明,DistDGL在不影响模型准确性的情况下实现了线性加速,并且在16台机器的集群上,仅需13秒即可完成具有1亿个节点和30亿条边的图的训练时期。DistDGL现在作为DGL的一部分公开提供:https://github.com/dmlc/dgl/tree/master/python/dgl/distributed。

英文摘要:

Graph neural networks (GNN) have shown great success in learning from graph-structured data. They are widely used in various applications, such as recommendation, fraud detection, and search. In these domains, the graphs are typically large, containing hundreds of millions of nodes and several billions of edges. To tackle this challenge, we develop DistDGL, a system for training GNNs in a mini-batch fashion on a cluster of machines. DistDGL is based on the Deep Graph Library (DGL), a popular GNN development framework. DistDGL distributes the graph and its associated data (initial features and embeddings) across the machines and uses this distribution to derive a computational decomposition by following an owner-compute rule. DistDGL follows a synchronous training approach and allows ego-networks forming the mini-batches to include non-local nodes. To minimize the overheads associated with distributed computations, DistDGL uses a high-quality and light-weight min-cut graph partitioning algorithm along with multiple balancing constraints. This allows it to reduce communication overheads and statically balance the computations. It further reduces the communication by replicating halo nodes and by using sparse embedding updates. The combination of these design choices allows DistDGL to train high-quality models while achieving high parallel efficiency and memory scalability. We demonstrate our optimizations on both inductive and transductive GNN models. Our results show that DistDGL achieves linear speedup without compromising model accuracy and requires only 13 seconds to complete a training epoch for a graph with 100 million nodes and 3 billion edges on a cluster with 16 machines. DistDGL is now publicly available as part of DGL:https://github.com/dmlc/dgl/tree/master/python/dgl/distributed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑