arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.23918cs.DCcs.LGcs.PF
收藏

模型停放税:量化始终在线GPU模型部署的隐藏能耗成本

The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment

Sai Sathvik Vadari

AI总结:

通过跨架构测量,发现GPU空闲功耗由DVFS状态决定,而非显存占用,CUDA上下文贡献了超过98%的停放税,并建立了冷启动盈亏平衡模型。

摘要:

AI推理行业将模型全天候加载在GPU内存中以避免冷启动延迟,隐含地将空闲功耗视为准备就绪的固定成本。然而,这种成本的结构从未被经验性地分解——也从未跨GPU架构进行过。我们首次跨架构测量了空闲GPU功耗作为VRAM分配的函数,结合了18天的生产遥测数据(335,267个样本,14个H100 GPU)以及在三种GPU架构(涵盖三种内存技术:NVIDIA H100(HBM3,80 GB)、A100(HBM2e,80 GB)和L40S(GDDR6,48 GB))上进行的受控剂量-反应实验。我们观察到,在所有三种架构上,空闲功耗是分段常数:CUDA上下文强制进行离散的DVFS转换,消耗比裸空闲多26-66 W(HBM架构上为26-50 W,GDDR6上为66 W),而边际VRAM效应在所有测试设备上均低于测量相关性(|β| < 0.02 W/GB)。无论内存技术如何,CUDA上下文占停放税的98%以上。我们通过在所有三种架构上运行真实的HuggingFace模型(Qwen2.5-7B)验证了这一发现,确认每个设备上空张量与模型加载之间的功耗差异小于0.5 W,并捕获了模型加载期间的冷启动功耗曲线。我们推导出一个冷启动盈亏平衡模型,表明能量最优行为取决于请求到达率和加载延迟——而非模型大小——盈亏平衡间隔为1-5分钟。我们的结果确定了一个在所有测试架构上一致的约束:带上下文的空闲功耗由DVFS状态决定,而非内存占用。

英文摘要:

The AI inference industry keeps models loaded in GPU memory around the clock to avoid cold-start latency, implicitly treating idle power as a fixed cost of readiness. Yet the structure of this cost has never been empirically decomposed - and never across GPU architectures. We present the first cross-architecture measurement of idle GPU power as a function of VRAM allocation, combining 18 days of production telemetry (335,267 samples, 14 H100 GPUs) with controlled dose-response experiments on three GPU architectures spanning three memory technologies: NVIDIA H100 (HBM3, 80 GB), A100 (HBM2e, 80 GB), and L40S (GDDR6, 48 GB). We observe that idle power is piecewise constant on all three architectures: the CUDA context forces a discrete DVFS transition consuming +26-66 W over bare idle (26-50 W on HBM architectures, 66 W on GDDR6), while the marginal VRAM effect is bounded below measurement relevance ($|β| < 0.02$ W/GB) on every device tested. The CUDA context accounts for >98% of the parking tax regardless of memory technology. We validate this finding with a real HuggingFace model (Qwen2.5-7B) on all three architectures, confirming <0.5 W difference from empty tensors on every device, and capture cold-start power profiles during model loading. We derive a cold-start breakeven model showing energy-optimal behavior depends on request arrival rate and loading latency - not model size - with breakeven intervals of 1-5 minutes. Our results identify a constraint consistent across all tested architectures: idle-with-context power is determined by DVFS state, not memory occupancy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑