arXiv 2109.078932021-09-16cs.DCcs.LG

Efficient Scaling of Dynamic Graph Neural Networks

IBM Research, India（IBM印度研究院）；University of Utah（犹他大学）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）；University of Tokyo（东京大学）

Venkatesan T. Chakaravarthy, Shivmaran S. Pandian, Saurabh Raje, Yogish Sabharwal, Toyotaro Suzumura, Shashanka Ubaru

备注：: Conference version to appear in the proceedings of SC'21
DOI：: 10.1145/3458817.3480858

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time. We develop a simple, but effective data distribution technique under which the communication volume remains fixed and linear in the input size, for any number of GPUs. Our experiments using billion-size graphs on a system of 128 GPUs shows that: (i) the distribution scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference technique reduces the transfer time by a factor of up to 4.1x and the overall execution time by up to 40%

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

迈向图上的神经缩放定律

Towards Neural Scaling Laws on Graphs

Michigan State University（密歇根州立大学）；Snap Inc（Snap公司）

该研究通过覆盖多任务、最高1亿参数和5000万样本的基准实验，建立并修正图模型与图数据的神经缩放定律。

2024-11-30cs.AIcs.LG

材料科学中的断裂神经缩放定律

Broken neural scaling laws in materials science

Institute of Physics（物理研究所）；Institute of Micro（微研究所）

本研究发现材料科学中预测介电函数的神经缩放定律在数据集规模上失效，但模型参数数量的缩放遵循简单幂律。

2026-02-05cond-mat.mtrl-scics.LG

Scaling Knowledge Graph Embedding Models

IBM Research Almaden（IBM阿尔马登研究中心）；Instacart（Instacart公司）

Developing scalable solutions for training Graph Neural Networks (GNNs) for link prediction tasks is challenging due to the high data dependencies which entail high computational cost and huge memory footprint. We propose a new method for scaling training of knowledge graph embedding models for link prediction to address these challenges. Towards this end, we propose the following algorithmic strategies: self-sufficient partitions, constraint-based negative sampling, and edge mini-batch training. Both, partitioning strategy and constraint-based negative sampling, avoid cross partition data tra…

2022-01-08cs.AIcs.LG

GETS：图神经网络校准中的集成温度缩放

GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks

Massachusetts Institute of Technology（麻省理工学院）；The Chinese University of Hong Kong（香港中文大学）；Singapore-MIT Alliance for Research and Technology (SMART)（新加坡-麻省理工研究与技术联盟）；University of Florida（佛罗里达大学）

针对图神经网络校准不佳的问题，提出GETS框架，结合输入与模型集成策略，在10个基准数据集上将期望校准误差降低25%。

2025-02-27cs.LG

STaR: Knowledge Graph Embedding by Scaling, Translation and Rotation

Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）

The bilinear method is mainstream in Knowledge Graph Embedding (KGE), aiming to learn low-dimensional representations for entities and relations in Knowledge Graph (KG) and complete missing links. Most of the existing works are to find patterns between relationships and effectively model them to accomplish this task. Previous works have mainly discovered 6 important patterns like non-commutativity. Although some bilinear methods succeed in modeling these patterns, they neglect to handle 1-to-N, N-to-1, and N-to-N relations (or complex relations) concurrently, which hurts their expressiveness. …

2022-02-15cs.LG

图Transformer的扩展：稀疏与密集注意力的比较研究

Scaling Graph Transformers: A Comparative Study of Sparse and Dense Attention

Technical University of Munich（慕尼黑技术大学）

针对传统图神经网络难以捕获节点长程依赖的问题，本文比较图Transformer中密集与稀疏两种注意力机制的权衡及适用场景，概述其设计挑战，为图Transformer的扩展提供参考。

2025-08-24cs.AIcs.LG

更多相关论文收起更多论文

Scaling up graph homomorphism for classification via sampling

LAMSADE, Université Paris-Dauphine, PSL（巴黎第九大学（多菲纳大学）LAMSADE实验室，PSL研究大学）

Feature generation is an open topic of investigation in graph machine learning. In this paper, we study the use of graph homomorphism density features as a scalable alternative to homomorphism numbers which retain similar theoretical properties and ability to take into account inductive bias. For this, we propose a high-performance implementation of a simple sampling algorithm which computes additive approximations of homomorphism densities. In the context of graph machine learning, we demonstrate in experiments that simple linear models trained on sample homomorphism densities can achieve per…

2021-04-08cs.DScs.LG

神经网络的集合变量：经验时间演化与标度律

Collective variables of neural networks: empirical time evolution and scaling laws

Institute for Computational Physics（计算物理研究所）；University of Stuttgart（斯图加特大学）；Cavendish Laboratory（卡文迪什实验室）；DAMTP（应用数学与理论物理系）；University of Cambridge（剑桥大学）；Institute for Particle Physics Phenomenology（粒子物理现象学研究所）；Durham University（杜伦大学）

本文提出通过经验神经正切核谱的熵和迹来理解神经网络学习动力学，发现信息压缩和结构形成两种普遍机制，并将结构形成视为深度学习阶段的起始标志。

2024-10-09cs.LGphysics.comp-ph

Graph Neural Processes：贝叶斯图神经网络

Graph Neural Processes: Towards Bayesian Graph Neural Networks

我们介绍了图神经过程（GNP），其灵感来自于最近在条件和潜在神经过程方面的工作。图神经过程被定义为对任意图数据进行操作的条件神经过程。它将稀疏观察的上下文点的特征作为输入，并输出目标点上的分布。我们展示了图神经过程中的边缘插补，并讨论了其他应用领域的方法的优点和缺点。GNP的一个主要好处是能够量化图结构上的深度学习中的不确定性。该方法的另一个好处是能够将图神经网络扩展到动态大小的图的输入。

2019-10-01cs.LGstat.ML

基于长短期记忆循环神经网络的云资源自动伸缩算法

Automatic Cloud Resource Scaling Algorithm based on Long Short-Term Memory Recurrent Neural Network

可伸缩性是云计算的一个重要特征。通过可扩展性，可以根据需求调配和释放资源，从而最大限度地降低成本。当前大多数基础设施即服务（IaaS）提供商都提供基于阈值的自动扩展技术。然而，设置具有正确值的阈值以最小化成本并实现服务水平协议并不是一件容易的事情，特别是在工作负载变化和突然变化的情况下。本文提出了基于动态阈值的自动缩放算法，该算法使用长短期记忆循环神经网络预测所需的资源，并根据预测值自动缩放虚拟资源。所提出的算法进行了评估，并与现有的一些算法进行了比较。实验结果表明，该算法的性能优于其他算法.

2017-01-12cs.DC

优化分布式训练方法以扩展神经网络规模

Optimizing Distributed Training Approaches for Scaling Neural Networks

本文比较了数据并行、模型并行和混合并行策略，提出自适应调度算法，在CIFAR-100上实现3.2倍加速和额外18%效率提升。

2025-03-29cs.DC

CarbonScaling：扩展神经缩放定律以用于大型语言模型中的碳足迹

CarbonScaling: Extending Neural Scaling Laws for Carbon Footprint in Large Language Models

Indiana University（印第安纳大学）

本文提出CarbonScaling框架，结合神经缩放定律和分布式训练策略，准确建模前沿LLM训练的碳足迹，提高硬件配置和排放估计的精度。

2026-05-15cs.AIcs.CLcs.CY

扩展到包括多达20 G突触的脉冲神经网络的分布式仿真的1024个软件进程和硬件核心

Scaling to 1024 software processes and hardware cores of the distributed simulation of a spiking neural network including up to 20G synapses

这篇简短的报告描述了塑料脉冲神经网络的分布式模拟器的扩展，高达1024个软件过程和硬件核心。以前的报告表明，模拟器具有良好的可扩展性，最多可达128个进程。在这里，我们扩展到1和1024个软件进程和硬件核心之间的范围内的模拟器的加速测量和强弱缩放分析。我们模拟了皮质柱的二维网格，包括连接~ 11 M神经元的~ 20 G突触。神经网络分布在一组MPI进程上，模拟在由多达64个双插槽节点组成的服务器平台上运行，每个插槽配备Intel Haswell E5-2630 v3处理器（8核@2.4 GHz时钟）。所有节点通过InfiniBand网络互连。DPSNN模拟器由INFN在EURETILE和CORTICONIC欧洲FET项目的框架内开发，并将在人脑项目（HBP）子项目2 -认知和系统神经科学的框架内由WaveScalEW tem使用。本报告为与神经模拟工具NEST进行更全面的比较奠定了基础。

2015-11-30cs.DCq-bio.NC

Time-Efficient and High-Quality Graph Partitioning for Graph Dynamic Scaling

The dynamic scaling of distributed computations plays an important role in the utilization of elastic computational resources, such as the cloud. It enables the provisioning and de-provisioning of resources to match dynamic resource availability and demands. In the case of distributed graph processing, changing the number of the graph partitions while maintaining high partitioning quality imposes serious computational overheads as typically a time-consuming graph partitioning algorithm needs to execute each time repartitioning is required. In this paper, we propose a dynamic scaling method tha…

2021-01-18cs.DBcs.DCcs.DM