arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.12689cs.LG
收藏

IC-Cache: 通过上下文缓存实现高效的大语言模型服务

IC-Cache: Efficient Large Language Model Serving via In-context Caching

University of Illinois Urbana-Champaign;Google;Google \& University of Washington

Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David Culler

AI总结:

针对大语言模型大规模服务的高资源需求与高延迟问题,提出IC-Cache缓存系统,通过复用大模型历史请求-回复对增强小模型能力,结合自适应路由与成本感知缓存优化,在保证回复质量的同时大幅提升吞吐量、降低延迟。

摘要:

大语言模型(LLM)已在各类应用中表现出色,但由于其巨大的资源需求和高延迟,大规模部署服务颇具挑战。我们的实际研究显示,超过70%的LLM用户请求存在语义相似的对应请求,这表明请求之间存在知识迁移的潜力。然而,简单地缓存并复用过往回复会导致质量大幅下降。本文提出IC-Cache,一种可实现LLM能力实时增强以提升服务效率的缓存系统:通过将来自更大模型的历史请求-回复对作为上下文示例,IC-Cache使小型LLM能够模仿甚至超越更大模型的组合能力(如推理能力),从而实现请求的选择性卸载,以降低成本和延迟。要大规模实现这种实时增强,需要在回复质量、延迟和系统吞吐量之间进行复杂的权衡。对于新请求,IC-Cache会高效筛选出相似的高效用示例,将其附加到新请求的输入之前。在大规模场景下,它会综合考虑回复质量和服务负载,在不同能力的LLM之间自适应路由请求。IC-Cache采用成本感知的缓存重放机制,可在离线状态下优化示例质量,以最大化在线缓存的效用和效率。针对数百万条真实请求的评估表明,IC-Cache将LLM服务吞吐量提升了1.4-5.9倍,延迟降低了28-71%,且不会损害回复质量。

英文摘要:

Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 70% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge transfer among requests. However, naively caching and reusing past responses leads to a big quality drop. In this paper, we introduce IC-Cache, a caching system that enables live LLM capability augmentation to improve serving efficiency: by leveraging historical request-response pairs from larger models as in-context examples, IC-Cache empowers small LLMs to imitate and even exceed the compositional abilities (e.g., reasoning) of their larger counterparts, enabling selective offloading of requests to reduce cost and latency. Achieving this live augmentation at scale introduces intricate trade-offs between response quality, latency, and system throughput. For a new request, IC-Cache efficiently selects similar, high-utility examples to prepend them to the new request's input. At scale, it adaptively routes requests across LLMs of varying capabilities, accounting for response quality and serving loads. IC-Cache employs a cost-aware cache replay mechanism that refines example quality offline to maximize online cache utility and efficiency. Evaluations on millions of realistic requests demonstrate that IC-Cache improves LLM serving throughput by 1.4-5.9x and reduces latency by 28-71% without hurting response quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑