arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04585cs.LGcs.AIcs.DC
收藏

CubicML:基于性能的ML预测实现大型ML系统与算法协同设计的自动化机器学习

CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performance

AI at Meta(Meta人工智能部门)

Wei Wen, Quanyu Zhu, Weiwei Chu, Wen-Yen Chen, Jiyan Yang

AI总结:

本文提出CubicML框架,利用ML模型作为代理预测训练性能,自动优化大型分布式ML系统的训练速度,有效提升了730亿参数广告推荐模型和4050亿参数大语言模型的训练性能。

摘要:

扩大深度学习模型规模已被证明能有效提升机器学习(ML)模型的智能水平,特别是对于工业推荐模型和大型语言模型。大型分布式ML系统与算法的协同设计(以最大化训练性能)对其成功起着关键作用。随着规模扩大,协同设计超参数的数量迅速增长,这为可行地寻找最优设置以最大化系统性能带来了挑战。在本文中,我们提出了CubicML,它利用ML自动优化大型分布式ML系统的训练性能。在CubicML中,我们使用一个ML模型作为代理来预测训练性能,以实现搜索效率和性能建模的灵活性。我们证明,CubicML能够有效优化Meta内部拥有730亿参数的广告推荐模型以及多达4050亿参数的大型语言模型的训练速度。

英文摘要:

Scaling up deep learning models has been proven effective to improve intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of large distributed ML systems and algorithms (to maximize training performance) plays a pivotal role for its success. As it scales, the number of co-design hyper-parameters grows rapidly which brings challenges to feasibly find the optimal setup for system performance maximization. In this paper, we propose CubicML which uses ML to automatically optimize training performance of large distributed ML systems. In CubicML, we use an ML model as a proxy to predict the training performance for search efficiency and performance modeling flexibility. We proved that CubicML can effectively optimize training speed of in-house ads recommendation models with 73 billion parameters and large language models up to 405 billion parameters at Meta.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑