arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.02600cs.DCcs.AI
收藏

TAPAS:云平台中面向 LLM 推理的热与功耗感知调度

TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms

Microsoft Azure Research(微软 Azure 研究院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas, Ricardo Bianchini

AI总结:

针对云数据中心 LLM 推理的毫秒级热、功耗波动及 SaaS/IaaS 混合部署问题,提出 TAPAS 框架,通过 VM 放置、请求路由和动态重配置减少节流事件并降低 TCO。

摘要:

生成式大型语言模型(LLM)需求的不断增长,给云数据中心的热管理和功耗管理带来了挑战。传统技术通常不足以应对 LLM 推理,因为其具有细粒度、毫秒级的执行阶段,每个阶段都有不同的性能、热和功耗特征。此外,LLM 推理工作负载对多种配置参数(例如模型并行度、模型规模和量化)敏感,这些参数涉及性能、温度、功耗与输出质量之间的权衡。而且,云环境通常将 SaaS 和 IaaS 工作负载协同部署,二者具有不同程度的可见性和灵活性。我们提出 TAPAS,这是一个面向云中 LLM 推理集群的热与功耗感知框架。TAPAS 增强制冷和电力超额订阅能力,在有效处理紧急情况(例如制冷和电力故障)的同时降低总拥有成本(TCO)。该系统利用历史温度和功耗数据以及 SaaS 工作负载的适应性,以:(1)在制冷和功耗约束内高效放置新的 GPU 工作负载 VM;(2)跨 SaaS VM 路由 LLM 推理请求;(3)重新配置 SaaS VM,以管理负载峰值和紧急情况。我们在一个大型 GPU 集群上的评估表明,热节流和功耗节流事件显著减少,从而提升了系统效率。

英文摘要:

The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques often are inadequate for LLM inference due to the fine-grained, millisecond-scale execution phases, each with distinct performance, thermal, and power profiles. Additionally, LLM inference workloads are sensitive to various configuration parameters (e.g., model parallelism, size, and quantization) that involve trade-offs between performance, temperature, power, and output quality. Moreover, clouds often co-locate SaaS and IaaS workloads, each with different levels of visibility and flexibility. We propose TAPAS, a thermal- and power-aware framework designed for LLM inference clusters in the cloud. TAPAS enhances cooling and power oversubscription capabilities, reducing the total cost of ownership (TCO) while effectively handling emergencies (e.g., cooling and power failures). The system leverages historical temperature and power data, along with the adaptability of SaaS workloads, to: (1) efficiently place new GPU workload VMs within cooling and power constraints, (2) route LLM inference requests across SaaS VMs, and (3) reconfigure SaaS VMs to manage load spikes and emergency situations. Our evaluation on a large GPU cluster demonstrates significant reductions in thermal and power throttling events, boosting system efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑