高效、受限于VRAM的xLM客户端推断
Efficient, VRAM-Constrained xLM Inference on Clients
AI总结:
本文提出pipelined sharding技术,通过CPU-GPU混合调度优化xLM在客户端的高效推断,提升TTFT和TPS性能,适用于LLM和VLM。
摘要:
为推动下一代客户端AI创新,迫切需要在客户端系统上实现高效、无损的高精度大语言模型(LLMs)和视觉语言模型(VLMs)的推理,统称为xLMs。为此,我们提出了pipelined sharding技术,一种新颖的、基于基准测试的CPU-GPU混合调度技术,以实现高效、受VRAM限制的LLM推理。通过结合子层模型分片、CPU卸载、流水线复制计算以及优先级张量放置,它优化了时间到第一个令牌(TTFT)和每秒令牌数(TPS)指标,并灵活适应系统和推理条件。对于高效、高精度的VLM推理,我们结合流水线分片与llama$.$cpp实现的三种已知前序思想(统称为VLMOpt),即视觉张量CPU卸载、flash注意力以及视觉和语言模型VRAM重叠避免。这些增强旨在改进未来两个重要NVIDIA产品——In-Game Inferencing软件开发工具包(IGI SDK)和Cosmos-Reason1(CR1)物理AI推理VLM的客户端xLM推理。我们的严格评估涵盖多个模型和客户端系统,亮点包括:在交互使用中,LLM的TTFT提升最高达6.7倍,TPS提升最高达30倍,而CR1推理的VRAM需求降低10倍;在批量模式下,吞吐量提升最高达8.2倍,均与各自激进的基线相比。本文被第9届MLSys会议(行业跟踪)接受,2026年。代码和成果可在:https://github.com/deepshnv/pipeshard-mlsys26-ae获取。
英文摘要:
To usher in the next round of client AI innovation, there is an urgent need to enable efficient, lossless inference of high-accuracy large language models (LLMs) and vision language models (VLMs), jointly referred to as xLMs, on client systems. To address this, we present pipelined sharding, a novel, benchmark-profile-guided CPU-GPU hybrid scheduling technique to achieve efficient, VRAM-constrained inference for both dense and mixture-of-experts (MoE) LLMs. Using a combination of model sharding at the sub-layer level, CPU offloading, pipelined copy-compute, and prioritized tensor placement in VRAM, it optimizes both time-to-first-token (TTFT) and tokens per second (TPS) metrics, while flexibly adapting to system and inference conditions. For efficient, high-accuracy VLM inference, we combine pipelined sharding with a llama$.$cpp implementation of three well-understood prior ideas (jointly called VLMOpt), namely, vision tensor CPU offloading, flash attention, and vision and language model VRAM overlap avoidance. These enhancements are targeted at improving client xLM inference in future releases of two important NVIDIA products - the In-Game Inferencing software development kit (IGI SDK) and the Cosmos-Reason1 (CR1) physical AI reasoning VLM. Highlights from our rigorous evaluation spanning multiple models and client systems include: for interactive use, TTFT improves by up to 6.7x and TPS by up to 30x for LLMs, and CR1 inference's VRAM demand is down by 10x, while in batched mode, throughput improves by up to 8.2x, all compared to their respective aggressive baselines. This paper is accepted at the 9th MLSys Conference (Industry Track), 2026. Code and artifact available at: https://github.com/deepshnv/pipeshard-mlsys26-ae
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。