并行与分布式图神经网络：深入的并发分析
Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis
AI总结：
本文构建GNN并行性分类体系，结合工作深度、通信与同步分析，系统评估模型、任务、框架和加速器，并推广消息传递流水线以指导高效并行GNN设计。
摘要：
图神经网络（GNN）是深度学习中最强大的工具之一。它们通常以高精度解决非结构化网络上的复杂问题，例如节点分类、图分类或链接预测。然而，GNN的推理和训练都很复杂，并且它们以独特方式将非规则图处理的特征与密集且规则的计算结合起来。这种复杂性使得在现代大规模并行架构上高效执行GNN极具挑战性。为缓解这一问题，我们首先设计了一种GNN中并行性的分类体系，考虑数据并行与模型并行，以及不同形式的流水线。随后，我们使用该分类体系研究众多GNN模型、由GNN驱动的机器学习任务、软件框架或硬件加速器中的并行性程度。我们采用work-depth（工作深度）模型，并评估通信量与同步。我们特别关注相关张量的稀疏性/稠密性，以理解如何有效应用向量化等技术。我们还形式化分析了GNN流水线，并将已确立的Message-Passing（消息传递）类GNN模型推广到覆盖任意流水线深度，从而促进未来优化。最后，我们研究不同形式的异步性，为未来的异步并行GNN流水线指明道路。我们的分析结果被综合为一组有助于最大化GNN性能的洞见，以及一份关于高效GNN计算进一步研究的挑战与机遇的综合清单。我们的工作将有助于推动未来GNN设计的发展。
英文摘要：
Graph neural networks (GNNs) are among the most powerful tools in deep learning. They routinely solve complex problems on unstructured networks, such as node classification, graph classification, or link prediction, with high accuracy. However, both inference and training of GNNs are complex, and they uniquely combine the features of irregular graph processing with dense and regular computations. This complexity makes it very challenging to execute GNNs efficiently on modern massively parallel architectures. To alleviate this, we first design a taxonomy of parallelism in GNNs, considering data and model parallelism, and different forms of pipelining. Then, we use this taxonomy to investigate the amount of parallelism in numerous GNN models, GNN-driven machine learning tasks, software frameworks, or hardware accelerators. We use the work-depth model, and we also assess communication volume and synchronization. We specifically focus on the sparsity/density of the associated tensors, in order to understand how to effectively apply techniques such as vectorization. We also formally analyze GNN pipelining, and we generalize the established Message-Passing class of GNN models to cover arbitrary pipeline depths, facilitating future optimizations. Finally, we investigate different forms of asynchronicity, navigating the path for future asynchronous parallel GNN pipelines. The outcomes of our analysis are synthesized in a set of insights that help to maximize GNN performance, and a comprehensive list of challenges and opportunities for further research into efficient GNN computations. Our work will help to advance the design of future GNNs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
