发表机构
Lenovo Research; University of Washington（联想研究院; 华盛顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对单GPU部署70B参数大语言模型的内存、吞吐量及集成成本问题，构建了含剪枝、量化、KV缓存压缩的预算感知压缩流水线，实现了模型压缩至33GB、57 tokens/s推理速度且精度损失在5%内的效果，提供了设计规则与评估协议。
AI 中文摘要
在NVIDIA GPU上部署70B参数大语言模型时，受限于设备内存、长上下文吞吐量及工程集成成本，我们将单GPU推理视为在这三个维度上的预算感知设计问题，研究剪枝、量化与KV缓存压缩在实际执行中的相互作用。控制消融实验表明，逐层剪枝可提升权重量化的鲁棒性；KV缓存稀疏化与INT8 KV量化互补，在不损害解码速度的前提下减少内存占用，而静态向量量化器常与动态缓存存在冲突。基于这些耦合结果与显式预算追踪，我们构建了一套实用流水线，将70B模型压缩至约33GB，在单张A40上处理10k token提示时维持约57 tokens/s的速度，在通用及推理基准上的绝对精度控制在5%以内。我们提供了设计规则与可复现的评估协议，该协议同时报告质量、内存与端到端速度，并为实际单GPU约束下的自动化流水线搜索奠定基础。
英文摘要
Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.
CommentsWithdrawn because the submission was made without the required authorization from a co-author for public disclosure
