通过记忆化和快进技术加速大型模型训练的分组级网络模拟
Supercharging Packet-level Network Simulation of Large Model Training via Memoization and Fast-Forwarding
AI总结:
Wormhole通过记忆化和快进技术显著提升大型模型训练的分组级网络模拟效率,实现高达1012倍的速度提升。
摘要:
分组级离散事件模拟(PLDES)是一种评估大型模型训练详细性能的常用工具。尽管PLDES提供了高保真度和通用性,但其缓慢的性能却困扰着网络研究者。现有优化技术要么简化网络模型,导致较大的误差;要么使用多个处理器并行执行,速度提升有上限。本文探索了一种替代的优化方向,通过减少PLDES的计算负载同时保持高保真度。我们的关键见解是,在分布式大语言模型训练中,分组级流量行为往往表现出重复的竞争模式和稳定状态,其中流量速率稳定。忽略这些冗余的离散事件可以显著加快模拟速度,误差可忽略不计。我们通过提出Wormhole,一种用户透明的PLDES内核,能够自动记忆不稳定状态并跳过稳定状态。Wormhole采用网络划分、状态记忆和重用以及基于速率的稳定状态识别,以准确确定每个流的稳定状态期间,同时在快进后保持模拟一致性。实验表明,Wormhole相比原始ns-3(MoE工作负载为510x)可实现744x的速度提升,误差小于1%。应用当前多线程并行技术与Wormhole结合,可实现1012x的速度提升,将128个GPU下一个GPT-13B训练的模拟时间从9小时减少到5分钟。
英文摘要:
Packet-level discrete-event simulation (PLDES) is a prevalent tool for evaluating detailed performance of large model training. Although PLDES offers high fidelity and generality, its slow performance has plagued networking practitioners. Existing optimization techniques either simplify the network model, resulting in large errors; or execute it in parallel using multiple processors, with an upper bound on speedup. This paper explores an alternative optimization direction that reduces the computational loads of PLDES while maintaining high fidelity. Our key insight is that, in distributed LLM training, packet-level traffic behaviors often exhibit repetitive contention patterns and steady-states where flow rates stabilize, ignoring these redundant discrete events speeds up the simulation considerably and the error is negligible. We realize this idea by proposing Wormhole, a user-transparent PLDES kernel capable of automatically memoization for unsteady-states and skipping for steady-states. Wormhole adopts network partitioning, state memoization and reuse, and rate-based steady-state identification to accurately determine the periods of each flow's steady-state, while maintaining simulation consistency after fast-forwarding. Experiments demonstrate that Wormhole can achieve a 744x speedup over the original ns-3 (510x for MoE workload), with a bounded error of <1%. Applying current multithreading parallel techniques and Wormhole together allows a 1012x speedup, reducing the simulation time for one GPT-13B training under 128 GPUs from 9 hours to 5 minutes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。