一种用于通信感知评估流水线并行LLM训练的表格调度抽象
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
AI总结:
本文提出一种表格调度抽象和统一的多抽象方法,通过公式推理、理想化调度表和通信感知执行模拟,比较了GPipe、1F1B、Chimera和Hanayo等流水线调度方案,发现通信会抵消气泡分析的结构优势,调度排名依赖于执行环境。
摘要:
流水线并行是大型语言模型分布式训练的关键技术,因为它减少了每设备的参数和激活内存。然而,比较流水线调度方案是困难的:分析模型暴露了诸如气泡比率等结构量,而端到端硬件实验成本高昂且依赖于系统。在这项工作中,我们引入了一种表格调度抽象和一种统一的多抽象方法,该方法连接了基于公式的推理、理想化调度表和通信感知执行模拟。使用这个框架,我们在多个建模的系统配置下比较了GPipe、1F1B、Chimera和受限模式下的Hanayo。我们的结果表明,调度排名不是抽象不变的:通信可以抵消仅由气泡分析所暗示的结构优势。在本文考虑的假设下,GPipe和1F1B在运行时等价,但1F1B实现了更低的激活内存峰值。Chimera主要在低微批次数和通信友好的情况下具有优势,而Hanayo在其预期的受限工作点有效,但对网络瓶颈敏感。我们进一步研究了一种非对称的Chimera式放置,它没有减少全局峰值内存需求,但在浅流水线中显示出有限的运行时收益。总体而言,流水线调度质量仅在建模的执行环境背景下才有意义。
英文摘要:
Pipeline parallelism is a key technique for distributed training of large language models because it reduces per-device parameter and activation memory. However, comparing pipeline schedules is difficult: analytical models expose structural quantities such as bubble ratios, while end-to-end hardware experiments are costly and system-specific. In this work, we introduce a tabular schedule abstraction and a unified multi-abstraction methodology that connects formula-based reasoning, idealized schedule tables, and communication-aware execution simulation. Using this framework, we compare GPipe, 1F1B, Chimera, and Hanayo in its restricted regime across multiple modeled system configurations. Our results show that schedule rankings are not abstraction-invariant: communication can negate structural advantages suggested by bubble analysis alone. Under the assumptions considered here, GPipe and 1F1B are runtime-equivalent, but 1F1B achieves a lower activation-memory peak. Chimera is advantageous mainly at low microbatch counts and in communication-favorable regimes, while Hanayo is effective in its intended restricted operating point but remains sensitive to network bottlenecks. We further study an asymmetric Chimera-style placement, which does not reduce the global peak memory requirement but reveals limited runtime gains in shallow pipelines. Overall, pipeline schedule quality is meaningful only in the context of the modeled execution environment.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。