从精确命中到足够接近：LLM嵌入的语义缓存
From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings
AI总结：
本文提出语义缓存的离线和在线策略，通过多项式时间启发式方法提高LLM嵌入的语义准确性。
摘要：
大型语言模型（LLMs）的快速应用产生了对更快响应和更低成本的需求。通过嵌入实现的语义缓存，通过重用语义相似的请求来满足这一需求，但打破了传统缓存假设并提出了新的挑战。在本文中，我们探索了语义缓存的离线策略，证明实现最优的离线策略是NP难的，并提出了几种多项式时间启发式方法。我们还提出了在线语义感知的缓存策略，结合了近期性、频率和局部性。在多样化的数据集上的评估显示，尽管基于频率的策略是强有力的基线，但我们的新变种提高了语义准确性。我们的发现揭示了当前系统的有效策略，并突显了未来创新的大量潜力。所有代码都是开源的。
英文摘要：
The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
