FlexNPU: 用于动态LLM预填充-解码共置的透明NPU虚拟化
FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location
AI总结:
针对LLM服务中预填充与解码阶段资源需求互补的问题,提出FlexNPU,一种透明用户空间虚拟化层,通过动态共置调度提升吞吐量并降低TTFT。
摘要:
现代AI服务越来越依赖NPU进行常规推理和大语言模型服务。然而,当前的NPU部署通常将物理设备直接暴露给应用程序,这限制了对调度的运行时控制,并使得难以适应阶段级工作负载行为。这一限制在LLM服务中尤为明显,其中预填充阶段是计算密集型的,而解码阶段通常受限于内存带宽和KV缓存访问。静态预填充-解码(PD)分离减少了阶段干扰,但可能导致资源不平衡和不必要的数据移动。我们提出FlexNPU,一个用于昇腾NPU的透明用户空间虚拟化层。FlexNPU拦截AscendCL API,并通过每设备守护进程路由NPU操作,将未修改的应用程序与物理NPU设备解耦,无需修改模型代码、AI框架或NPU驱动程序。这一运行时边界使FlexNPU能够虚拟化NPU对象、控制算子调度,并支持LLM服务的阶段感知调度。特别是,FlexNPU实现了动态PD共置,根据预填充和解码互补的资源特性自适应调度。我们在华为昇腾NPU上实现了FlexNPU,并使用典型LLM工作负载进行评估。与直接NPU透传相比,FlexNPU没有引入可测量的推理开销,并在某些场景下略微提高了吞吐量。在384卡昇腾910C部署的DeepSeek-R1上,FlexNPU相比静态PD分离提高了5.15%和26.33%的吞吐量。在Qwen2.5-7B上,与静态PD共置相比,FlexNPU在保持可比吞吐量的同时,将TTFT降低了超过92%,且TPOT几乎不变。这些结果表明,透明NPU虚拟化是实现高效且响应迅速的LLM服务的实用基础。
英文摘要:
Modern AI serving increasingly relies on NPUs for conventional inference and large language model serving. However, current NPU deployments commonly expose physical devices directly to applications, which limits runtime control over scheduling and makes it difficult to adapt execution to phase-level workload behavior. This limitation is particularly evident in LLM serving, where the prefill phase is compute-intensive while the decode phase is often constrained by memory bandwidth and KV-cache accesses. Static prefill-decode (PD) disaggregation reduces phase interference, but can introduce resource imbalance and unnecessary data movement. We present FlexNPU, a transparent user-space virtualization layer for Ascend NPUs. FlexNPU interposes on AscendCL APIs and routes NPU operations through per-device daemons, decoupling unmodified from physical NPU devices without modifying model code, AI frameworks, or NPU drivers. This runtime boundary allows FlexNPU to virtualize NPU objects, control operator dispatch, and support phase-aware scheduling for LLM serving. In particular, FlexNPU enables dynamic PD co-location, which adapts scheduling between prefill and decode according to their complementary resource characteristics. We implement FlexNPU on Huawei Ascend NPUs and evaluate it with typical LLM workloads. Compared with direct NPU passthrough, FlexNPU introduces no measurable inference overhead and slightly improves throughput in some scenarios. On a 384-card Ascend 910C deployment of DeepSeek-R1, FlexNPU improves throughput over static PD disaggregation by 5.15% and 26.33%. On Qwen2.5-7B, compared with static PD co-location, FlexNPU maintains comparable throughput while reducing TTFT by over 92% across tested workloads with nearly unchanged TPOT. These results show that transparent NPU virtualization is a practical substrate for efficient and responsive LLM serving.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。