LLMem：估算用于微调预训练大语言模型的GPU显存使用量
LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMs
AI总结：
针对有限硬件下微调预训练LLM的GPU显存约束及最优分布式微调方法难确定的问题，提出LLMem方案，通过模型结构与方法显存分布提前估算显存并选最优方法，单卡误差最高1.6%，十亿参数以上多卡微调平均误差3.0%。
摘要：
在硬件有限的情况下微调预训练大语言模型（LLM）会因GPU显存限制面临挑战。学界已提出多种分布式微调方法来缓解GPU显存约束，但在给定环境中确定既能实现快速微调又能避免GPU显存不足问题的最优方法仍不明确。为解决这一难题，我们提出LLMem，该方案可估算在多GPU上应用分布式微调方法时的GPU显存消耗量，并找出最优方法。我们基于基于Transformer的解码器模型的基本结构以及每种方法的显存分布，在微调前开展GPU显存使用量估算。实验结果表明，LLMem能准确估算单GPU上的GPU显存峰值，误差率最高为1.6%。此外，在多GPU设置下对参数超过10亿的LLM应用分布式微调方法时，其平均误差率为3.0%。
英文摘要：
Fine-tuning pre-trained large language models (LLMs) with limited hardware presents challenges due to GPU memory constraints. Various distributed fine-tuning methods have been proposed to alleviate memory constraints on GPU. However, determining the most effective method for achieving rapid fine-tuning while preventing GPU out-of-memory issues in a given environment remains unclear. To address this challenge, we introduce LLMem, a solution that estimates the GPU memory consumption when applying distributed fine-tuning methods across multiple GPUs and identifies the optimal method. We conduct GPU memory usage estimation prior to fine-tuning, leveraging the fundamental structure of transformer-based decoder models and the memory usage distribution of each method. Experimental results show that LLMem accurately estimates peak GPU memory usage on a single GPU, with error rates of up to 1.6%. Additionally, it shows an average error rate of 3.0% when applying distributed fine-tuning methods to LLMs with more than a billion parameters on multi-GPU setups.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
