arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23556cs.LGcs.AIcs.DCcs.MAcs.PF
收藏

Rudder: 在分布式GNN训练中利用LLM代理进行转向预取

Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents

Iowa State University(爱荷华州立大学);Pacific Northwest National Laboratory(太平洋西北国家实验室);Amazon GenAI(亚马逊生成人工智能);University of California, Berkeley(加州大学伯克利分校)

Aishwarya Sarkar, Sayan Ghosh, Nathan Tallent, Aman Chadha, Tanya Roosta, Ali Jannesari

AI总结:

Rudder利用LLM代理的生成式AI能力,通过自适应预取减少分布式GNN训练中的通信开销,提升训练性能达91%。

摘要:

大规模图神经网络(GNNs)通常通过采样顶点的邻居到固定距离来训练。由于大规模输入图是分布式的,训练需要频繁的不规则通信,这会阻碍前进进度。此外,获取的数据会随着图、图分布、采样和批量参数以及缓存策略而变化。因此,任何静态预取方法都会错过适应不同动态条件的关键机会。在本文中,我们介绍了Rudder,这是一个嵌入在最新AWS DistDGL框架中的软件模块,用于自主预取远程节点并减少通信。Rudder的适应性与标准启发式方法和传统机器学习分类器形成对比。我们发现当代大型语言模型(LLMs)中发现的生成式AI表现出如零样本任务中的上下文学习(ICL)等涌现特性,具有逻辑多步推理能力。我们发现这种行为非常适合适应性控制,即使有相当大的欠训练。使用标准数据集和在NERSC Perlmutter超算上的未见过的配置进行评估,结果显示相对于基线DistDGL(无预取)的端到端训练性能提高了91%,相对于静态预取提高了82%,通信减少了超过50%。我们的代码可在https://github.com/aishwaryyasarkar/rudder-llm-agent上获得。

英文摘要:

Large-scale Graph Neural Networks (GNNs) are typically trained by sampling a vertex's neighbors to a fixed distance. Because large input graphs are distributed, training requires frequent irregular communication that stalls forward progress. Moreover, fetched data changes with graph, graph distribution, sample and batch parameters, and caching polices. Consequently, any static prefetching method will miss crucial opportunities to adapt to different dynamic conditions. In this paper, we introduce Rudder, a software module embedded in the state-of-the-art AWS DistDGL framework, to autonomously prefetch remote nodes and minimize communication. Rudder's adaptation contrasts with both standard heuristics and traditional ML classifiers. We observe that the generative AI found in contemporary Large Language Models (LLMs) exhibits emergent properties like In-Context Learning (ICL) for zero-shot tasks, with logical multi-step reasoning. We find this behavior well-suited for adaptive control even with substantial undertraining. Evaluations using standard datasets and unseen configurations on the NERSC Perlmutter supercomputer show up to 91% improvement in end-to-end training performance over baseline DistDGL (no prefetching), and an 82% improvement over static prefetching, reducing communication by over 50%. Our code is available at https://github.com/aishwaryyasarkar/rudder-llm-agent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑