arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.18009cs.DC
收藏

单GPU视角下的分布式训练:用于分布式训练的RankLens

A Distributed View from a Single GPU: RankLens for Distributed Training

Beihang University(北京航空航天大学)

Yufan Wang, Yihao Zhang, Yi Liu

AI总结:

RankLens是一种单GPU运行的分布式训练仿真系统,可在保留分布式语义的前提下,将分布式训练分析的资源需求降至单GPU,实现高通信结构匹配度与低时间误差。

摘要:

现代分布式训练的调试、调优与分析成本高昂,因为精准分析往往需要重复执行目标规模的任务。我们发现,许多分析任务依赖原始分布式执行路径,而非完整的全秩执行。我们提出RankLens,这是一种单秩仿真系统,可在单个GPU上运行一个幸存者秩,同时保留目标任务的分布式语义。该幸存者秩执行原始训练程序并进行真实的本地计算;在通信边界处,系统将真实的多秩通信替换为依赖任务的语义,选择性恢复已记录的通信结果以及已记录或估计的完成时间,同时保留分析所需的异步依赖关系。这使得能够从真实执行中重放,并支持使用现有调试器和性能分析工具进行分析、调试与调优。我们在DeepSpeed和Megatron-LM上针对8至64 GPU的目标配置对该系统进行评估:RankLens实现了100%的训练张量通信结构匹配,将仿真时间误差控制在5%以下,支持本地异常复现与执行分析,并将这些分析的资源需求从目标规模集群降至单个GPU。

英文摘要:

Modern distributed training is costly to debug, tune, and analyze because faithful analysis often requires repeated target-scale executions. We observe that many analysis tasks depend on the original distributed execution path, but not on full all-rank execution. We present RankLens, a single-rank emulation system that runs one survivor rank on a single GPU while preserving the target job's distributed semantics. The survivor rank executes the original training program and performs real local computation. At communication boundaries, the system replaces real multi-rank communication with task-dependent semantics, selectively restoring recorded communication results and recorded or estimated completion timing while preserving the asynchronous dependencies required for analysis. This enables replay from real executions and supports analysis, debugging, and tuning with existing debuggers and performance analysis tools. We evaluate the system on DeepSpeed and Megatron-LM across 8-64-GPU target configurations. RankLens achieves 100% training-tensor communication-structure match, keeps emulation timing error below 5%, supports local anomaly reproduction and execution analysis, and reduces the resource requirement of these analyses from target-scale clusters to a single GPU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑