arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03065cs.LG
收藏

计算还是加载KV缓存?何不两者兼得?

Compute Or Load KV Cache? Why Not Both?

University of Michigan(密歇根大学)

Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao

AI总结:

针对长上下文大语言模型推理中前缀缓存I/O带宽受限、预填充KV缓存计算开销高的瓶颈,提出并行利用计算与I/O资源的Cake调度系统,实现平均2.6倍的首令牌生成时间降低,提升长上下文推理效率。

摘要:

大语言模型(LLM)正越来越多地被部署在大规模在线服务中,支撑各类复杂应用。然而,预填充阶段生成键值(KV)缓存的计算开销构成了主要瓶颈,对于长上下文输入而言尤为突出。前缀缓存通过存储KV缓存以供复用来缓解这一问题,减少了冗余计算。尽管有其优势,前缀缓存却因存储设备I/O带宽有限而存在高延迟问题,制约了推理效率。为应对这一挑战,我们提出了Cake——一种新型KV缓存加载系统,可并行优化利用计算与I/O两类资源。Cake采用双向调度策略,动态平衡KV缓存的计算与加载过程,确保资源高效利用。此外,Cake还集成了自适应调度机制,可与无前缀缓存的请求无缝融合,提升系统吞吐量并适配波动的资源可用性。通过在不同硬件配置、数据集和存储条件下开展的大量评估,Cake相比纯计算和纯I/O方法,平均将首令牌生成时间(TTFT)降低了2.6倍。研究结果表明,Cake是优化长上下文LLM推理的有效且实用的方案,填补了大规模AI部署中计算效率与I/O效率之间的空白。

英文摘要:

Large Language Models (LLMs) are increasingly deployed in large-scale online services, enabling sophisticated applications. However, the computational overhead of generating key-value (KV) caches in the prefill stage presents a major bottleneck, particularly for long-context inputs. Prefix caching mitigates this issue by storing KV caches for reuse, reducing redundant computation. Despite its advantages, prefix caching suffers from high latency due to the limited I/O bandwidth of storage devices, constraining inference efficiency. To address this challenge, we introduce Cake, a novel KV cache loading system that optimally utilizes both computational and I/O resources in parallel. Cake employs a bidirectional scheduling strategy that dynamically balances KV cache computation and loading, ensuring efficient resource utilization. Additionally, Cake incorporates an adaptive scheduling mechanism that seamlessly integrates with non-prefix caching requests, improving system throughput and adapting to fluctuating resource availabilty. Through extensive evaluations across various hardware configurations, datasets, and storage conditions, Cake achieves on average 2.6x reduction in Time to First Token (TTFT) compared to compute-only and I/O-only methods. Our findings highlight Cake as an effective and practical solution for optimizing long-context LLM inference, bridging the gap between computation and I/O efficiency in large-scale AI deployments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑