DPS：基于半统一内存的双精度LLM服务系统
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
浏览论文内容
中文总结 AI 辅助
DPS通过半统一内存将权重内存变为弹性资源，在KV压力下切换低精度变体以扩展缓存，提升吞吐量2.1-3.3倍，pass@1最高提升41个百分点。
中文摘要 AI 辅助
现有的LLM服务系统对KV缓存内存进行虚拟化和优化，但在整个执行过程中将模型权重内存视为固定不变。近期关于多精度模型表示的研究挑战了这一设计，允许单个存储模型同时支持全精度和低精度执行，使得有效权重占用空间依赖于运行时。这在突发工作负载下创造了机会，因为KV缓存需求的临时峰值往往决定吞吐量和SLO合规性。我们提出了DPS，一个双精度LLM服务系统，将权重内存转变为弹性资源：在正常负载下，DPS提供全精度模型服务；在KV压力下，它切换到嵌套的低精度变体，并将未使用的权重内存重新用于KV缓存块。DPS基于半统一内存（SUM）构建，该内存将权重区域划分为一个持久的低精度子区域和一个在残差权重张量与KV缓存块之间交替的共享区域，同时保持与分页KV缓存管理的兼容性。我们在vLLM之上实现了DPS，并在密集模型和MoE模型以及各种生产工作负载轨迹上进行了评估。我们的结果表明，DPS相比静态FP16，将持续吞吐量提高了2.1至3.3倍，有效pass@1提高了最多41个百分点，同时保持了FP16级别的精度。
英文摘要
Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that \sysname improves sustained throughput by $2.1$--$3.3\times$ and effective pass@1 by up to $+41$\,pp over Static FP16, while preserving FP16-class accuracy.
发表机构
- Seoul National University（首尔国立大学）
- Van-Lang Institute of Semiconductor Technology（文郎半导体技术研究院）
- Moreh Vietnam（Moreh越南公司）
- Moreh（Moreh公司）
机构由 AI 辅助整理，请以论文原文为准。
