arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.12465cs.DCcs.LGcs.PF
收藏

TaxBreak: 通过开销分解揭示LLM推理的隐藏成本

TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

Prabhu Vellaisamy, Shreesh Tripathi, Vignesh Natarajan, Surya Santhan Thenarasu, Shawn Blanton, John P. Shen

AI总结:

TaxBreak通过分解主机可见的编排开销,识别优化应针对软件栈还是设备侧工作负载,揭示LLM推理中隐藏的成本问题。

摘要:

大型语言模型(LLM)推理广泛应用于交互式助手和代理系统。在延迟敏感的部署中,推理时间可能由主机侧开销主导。现有方法通常仅暴露这种成本作为汇总残差或启动/队列指标,这往往不足以确定应优化哪个执行层。本文提出了TaxBreak,一种基于跟踪的方法,用于分解主机可见的编排开销为三个组成部分:框架翻译时间、CUDA库翻译时间和内核启动路径时间。我们在NVIDIA H100和H200系统上验证了TaxBreak,并利用它推导出我们提出的主机-设备平衡指数(HDBI),这是一个有界总结指数,将设备活跃执行与主机可见的编排联系起来。在代表性密集和专家混合工作负载中,无论是预填充还是解码,我们显示,汇总延迟、GPU空闲或有界比率单独可能掩盖主导的优化目标。TaxBreak相反区分了应优化软件栈开销的情况与应减少设备侧工作的情况。我们进一步表明,MoE模型每输出token调度的内核数量比密集模型多8-11倍,并且对于此类主机绑定工作负载,CPU单线程性能是首要参数:更快的主机CPU可减少编排开销10-29%,并提高端到端延迟最高14%,即使与较慢时钟的GPU配对。这些结果将TaxBreak定位为评估优化努力应针对软件栈还是设备侧工作负载的诊断工具。

英文摘要:

Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latency-sensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, CPU single-thread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by 10-29% and improves end-to-end latency by up to 14%, even when paired with a slower-clocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑