Graph neural networks (GNNs) have been demonstrated as a powerful tool for analyzing non-Euclidean graph data. However, the lack of efficient distributed graph learning systems severely hinders applications of GNNs, especially when graphs are big and GNNs are relatively deep. Herein, we present GraphTheta, the first distributed and scalable graph learning system built upon vertex-centric distributed graph processing with neural network operators implemented as user-defined functions. This system supports multiple training strategies and enables efficient and scalable big-graph learning on distributed (virtual) machines with low memory. To facilitate graph convolutions, GraphTheta puts forward a new graph learning abstraction named NN-TGAR to bridge the gap between graph processing and graph deep learning. A distributed graph engine is proposed to conduct the stochastic gradient descent optimization with a hybrid-parallel execution, and a new cluster-batched training strategy is supported. We evaluate GraphTheta using several datasets with network sizes ranging from small-, modest- to large-scale. Experimental results show that GraphTheta can scale well to 1,024 workers for training an in-house developed GNN on an industry-scale Alipay dataset of 1.4 billion nodes and 4.1 billion attributed edges, with a cluster of CPU virtual machines (dockers) of small memory each (5$\sim$12GB). Moreover, GraphTheta can outperform DistDGL by up to $2.02\times$, with better scalability, and GraphLearn by up to $30.56\times$. As for model accuracy, GraphTheta is capable of learning as good GNNs as existing frameworks. To the best of our knowledge, this work presents the largest edge-attributed GNN learning task in the literature.
On Parallel or Distributed Asynchronous Iterations with Unbounded Delays and Possible Out of Order Messages or Flexible Communication for Convex Optimization Problems and Machine Learning
LAAS-CNRS(LAAS-法国国家科学研究中心);Université de Toulouse(图卢兹大学);CNRS(法国国家科学研究中心)
John Curtin School of Medical Research, Australian National University(澳大利亚国立大学约翰·库廷医学研究学院);Department of Information Technology and Electrical Engineering, ETH Zürich(苏黎世联邦理工学院信息科技与电气工程系);Mathematical Data Science Centre, Australian National University(澳大利亚国立大学数学数据科学中心);Walter and Eliza Hall Institute(沃尔特和伊丽莎白·霍尔研究所)
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
U. Washington(华盛顿大学);CMU(卡内基梅隆大学);Stanford(斯坦福大学);NUS(新加坡国立大学);TuSimple(图森未来);NYU(纽约大学);Microsoft(微软);U. Alberta(阿尔伯塔大学);MIT(麻省理工学院);NYU Shanghai(上海纽约大学)
Flexible Coded Distributed Convolution Computing for Enhanced Straggler Resilience and Numerical Stability in Distributed CNNs
College of Computer Science, Chongqing University(重庆大学计算机学院);School of Electrical Engineering, Korea University(韩国大学电气工程学院);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);City University of Hong Kong(香港城市大学)
Purdue University(普渡大学);Lawrence Berkeley Lab(劳伦斯伯克利实验室);Yahoo! Research(雅虎研究院);University of Electronic Science and Technology of China(电子科技大学);University of Cambridge(剑桥大学)
On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks
University of Rome “La Sapienza”(罗马“萨皮恩扎”大学);Luleå University of Technology(吕勒奥理工大学);Research Institutes of Sweden(瑞典研究院);University of California, Berkeley(加州大学伯克利分校)
CityLearn v2: Energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities
The University of Texas at Austin(得克萨斯大学奥斯汀分校);Concordia University(康考迪亚大学);Politecnico di Torino(都灵理工大学);Polytechnic of Porto(波尔图理工学院);PassiveLogic(帕西夫逻辑公司);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Deep Reinforcement Learning Microgrid Optimization Strategy Considering Priority Flexible Demand Side
Changchun Institute of Technology(长春工程学院);National and Local Joint Engineering Research Center for Smart Distribution Network Measurement, Control and Safe Operation Technology(智能配电网测控与安全运行技术国家地方联合工程研究中心);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));Institute of Oceanographic Instrumentation(海洋仪器研究所)