SparPR:面向消费级硬件上资源受限的大语言模型推理的运行时KV缓存恢复机制
SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware
AI总结:
SparPR是一种无需修改模型权重和离线训练的KV缓存恢复机制,在消费级硬件上可降低LLM推理延迟、提升吞吐量,且针精度优于StreamingLLM和H2O。
摘要:
在消费级硬件上运行大语言模型(LLM)时,关键瓶颈在于键值(KV)缓存,该缓存会随每个解码的token增长,很快耗尽单个消费级GPU的内存和带宽。本文提出SparPR(稀疏部分重计算),一种不修改模型权重且无需离线训练的运行时KV缓存恢复机制。SparPR结合了三个在线决策:查询感知的键块选择,将历史块与当前查询进行比对;动态预算控制,估计每个块的复制和重计算成本以决定保留多少个块;以及按需局部重计算与成本感知回退,将保留的块合并为连续区间,并根据运行时成本比较在稀疏重计算与完整恢复之间切换。在FlexGen/FlexLLMGen上实现后,SparPR在单个消费级笔记本电脑(RTX 3070 Ti 8GB,Ryzen 9 6900HX,16GB RAM)上进行评估,在1k、2k、4k提示token时,分别将端到端延迟降低31%、52%、63%,吞吐量提高1.4倍至2.7倍,且在44分钟内完成8k token的工作负载,而FlexGen基线会超时。在固定缓存预算下,SparPR的针精度(Needle Accuracy)在每个保留比例下均显著高于StreamingLLM和H2O, ablation研究证实这三种机制协同作用。
英文摘要:
Running large language models (LLMs) on consumer hardware is bottlenecked by the key-value (KV) cache, which grows with every decoded token and quickly exhausts the memory and bandwidth of a single consumer GPU. We present SparPR (Sparse Partial Recomputation), a runtime KV-cache recovery mechanism that does not modify model weights and requires no offline training. SparPR couples three online decisions: Query-Aware key-block selection scores historical blocks against the current query; dynamic budget control estimates per-block copy and recompute costs to decide how many blocks to retain; and on-demand local recomputation with cost-aware fallback merges retained blocks into continuous spans and switches between sparse recomputation and full recovery according to a runtime cost comparison. Implemented on FlexGen/FlexLLMGen, SparPR is evaluated on a single consumer laptop (RTX 3070 Ti 8GB, Ryzen 9 6900HX, 16GB RAM). It cuts end-to-end latency by 31%, 52%, and 63% at 1k, 2k, and 4k prompt tokens respectively, with 1.4x-2.7x higher throughput, and completes an 8k-token workload in 44 minutes where the FlexGen baseline times out. Under a fixed cache budget, SparPR retains a Needle Accuracy that stays well above StreamingLLM and H2O at every keep ratio, and ablation studies confirm that the three mechanisms act jointly.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。