arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.09307cs.DCcs.AI
收藏

Lumos:面向大规模LLM训练的高效性能建模与估计

Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training

Cornell University(康奈尔大学);Meta;Massachusetts Institute of Technology(麻省理工学院)

Mingyu Liang, Hiwot Tadese Kassa, Wenyin Fu, Brian Coutinho, Louis Feng, Christina Delimitrou

AI总结:

针对分布式LLM训练优化难的问题,提出跟踪驱动工具包Lumos,可准确捕捉预测LLM执行行为,在512块H100 GPU集群上验证误差仅3.3%,还能高效探索新配置性能。

摘要:

分布式环境下训练LLM面临模型执行、部署系统的复杂性以及可配置策略的广阔空间等重大挑战。尽管存在多种优化技术,但在实践中实现高效率仍很困难。能有效刻画和预测模型行为的准确性能模型对于指导优化工作和系统级研究至关重要。我们提出Lumos,一个面向大规模LLM训练的跟踪驱动型性能建模与估计工具包,旨在准确捕捉和预测现代LLM的执行行为。我们在生产级ML集群上使用各种GPT-3变体对Lumos进行评估,该集群最多配备512块NVIDIA H100 GPU,结果表明,在不同模型和配置下,它能以平均仅3.3%的误差重放执行时间及其他运行时细节。此外,我们验证了其从现有跟踪中估计新设置性能的能力,有助于高效探索模型和部署配置。

英文摘要:

Training LLMs in distributed environments presents significant challenges due to the complexity of model execution, deployment systems, and the vast space of configurable strategies. Although various optimization techniques exist, achieving high efficiency in practice remains difficult. Accurate performance models that effectively characterize and predict a model's behavior are essential for guiding optimization efforts and system-level studies. We propose Lumos, a trace-driven performance modeling and estimation toolkit for large-scale LLM training, designed to accurately capture and predict the execution behaviors of modern LLMs. We evaluate Lumos on a production ML cluster with up to 512 NVIDIA H100 GPUs using various GPT-3 variants, demonstrating that it can replay execution time with an average error of just 3.3%, along with other runtime details, across different models and configurations. Additionally, we validate its ability to estimate performance for new setups from existing traces, facilitating efficient exploration of model and deployment configurations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑