arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05235cs.DCcs.AIcs.ARcs.LG
收藏

具有SLO意识的GPU频率缩放用于能源高效的LLM推理服务

SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving

ETH Zürich(苏黎世联邦理工学院);National Technical University of Athens(雅典国家技术大学)

Andreas Kosmas Kakolyris, Dimosthenis Masouros, Petros Vavaroutsos, Sotirios Xydis, Dimitrios Soudris

AI总结:

throttLL'eM通过预测KV缓存使用和批处理大小,利用机器学习模型减少GPU频率和实例规模,从而在满足SLOs的前提下降低LLM推理的能耗达43.8%并提升能效1.71倍。

摘要:

随着大型语言模型(LLM)的普及,其对耗电的GPU的依赖不断加剧能源需求,引发了环境和经济方面的担忧。推理是LLM工作负载的主要部分,对提供者来说是一个关键挑战:在保证服务级别目标(SLOs)以确保最佳用户体验的前提下,最小化能源成本。在本文中,我们提出了throttLL'eM框架,通过实例和GPU频率缩放来减少能耗同时满足SLOs。throttLL'eM具备预测未来KV缓存使用情况和批处理大小的机制。利用一个接收这些预测作为输入的机器学习(ML)模型,throttLL'eM在迭代级别管理性能,以减少频率和实例大小来满足SLOs。我们展示所提出的ML模型实现了大于0.97的R²分数,并且平均每个秒的预测误差少于1次迭代。在LLM推理跟踪上的实验结果表明,与NVIDIA的Triton服务器相比,throttLL'eM在满足SLOs的情况下实现了高达43.8%的能耗降低,以及至少1.71倍的能效提升。

英文摘要:

As Large Language Models (LLMs) gain traction, their reliance on power-hungry GPUs places ever-increasing energy demands, raising environmental and monetary concerns. Inference dominates LLM workloads, presenting a critical challenge for providers: minimizing energy costs under Service-Level Objectives (SLOs) that ensure optimal user experience. In this paper, we present \textit{throttLL'eM}, a framework that reduces energy consumption while meeting SLOs through the use of instance and GPU frequency scaling. \textit{throttLL'eM} features mechanisms that project future KV cache usage and batch size. Leveraging a Machine-Learning (ML) model that receives these projections as inputs, \textit{throttLL'eM} manages performance at the iteration level to satisfy SLOs with reduced frequencies and instance sizes. We show that the proposed ML model achieves $R^2$ scores greater than 0.97 and miss-predicts performance by less than 1 iteration per second on average. Experimental results on LLM inference traces show that \textit{throttLL'eM} achieves up to 43.8\% lower energy consumption and an energy efficiency improvement of at least $1.71\times$ under SLOs, when compared to NVIDIA's Triton server.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑