arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.04693cs.CLcs.AIcs.LG
收藏

理解大语言模型微调的性能并估算其成本

Understanding the Performance and Estimating the Cost of LLM Fine-Tuning

University of Michigan(密歇根大学);Georgia Institute of Technology(佐治亚理工学院);Intel Labs(英特尔实验室)

Yuchen Xia, Jiho Kim, Yuhan Chen, Haojie Ye, Souvik Kundu, Cong Hao, Nishil Talati

AI总结:

本文针对单GPU上基于稀疏混合专家的大语言模型微调展开研究,分析其准确率与运行时特征,提出优化MoE层是性能提升关键,并开发验证了可估算云端微调成本的分析模型,助力从业者预算规划。

摘要:

由于训练大语言模型(LLM)成本过高,微调已成为一种颇具吸引力的替代方案,可借助有限的计算资源以经济高效的方式让LLM适配特定任务。本文对基于稀疏混合专家(MoE)的LLM微调进行特征分析,以了解其在单GPU上的准确率和运行时性能。我们的评估为稀疏版和稠密版MoE模型的训练效能,以及它们的运行时特征(包括最大批量大小、执行时间分解、端到端吞吐量、GPU硬件利用率和负载分布)提供了独特洞见。研究发现,MoE层的优化是进一步提升LLM微调性能的关键。利用我们的性能剖析结果,我们还开发并验证了一个用于估算云端LLM微调成本的分析模型。该模型基于模型参数和GPU架构,可估算LLM的吞吐量和训练成本,帮助工业界和学术界的从业者为特定模型的微调制定成本预算。

英文摘要:

Due to the cost-prohibitive nature of training Large Language Models (LLMs), fine-tuning has emerged as an attractive alternative for specializing LLMs for specific tasks using limited compute resources in a cost-effective manner. In this paper, we characterize sparse Mixture of Experts (MoE) based LLM fine-tuning to understand their accuracy and runtime performance on a single GPU. Our evaluation provides unique insights into the training efficacy of sparse and dense versions of MoE models, as well as their runtime characteristics, including maximum batch size, execution time breakdown, end-to-end throughput, GPU hardware utilization, and load distribution. Our study identifies the optimization of the MoE layer as crucial for further improving the performance of LLM fine-tuning. Using our profiling results, we also develop and validate an analytical model to estimate the cost of LLM fine-tuning on the cloud. This model, based on parameters of the model and GPU architecture, estimates LLM throughput and the cost of training, aiding practitioners in industry and academia to budget the cost of fine-tuning a specific model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑