arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.13165cs.IRcs.AI
收藏

异步验证语义缓存用于分层大语言模型架构

Asynchronous Verified Semantic Caching for Tiered LLM Architectures

Asmit Kumar Singh, Haozhe Wang, Laxmi Naga Santosh Attaluri, Tak Chiam, Weihua Zhu

AI总结:

本文提出Krites异步缓存策略,通过LLM判断扩展静态缓存覆盖范围,提升静态缓存命中率,减少推理成本和延迟。

摘要:

大型语言模型(LLMs)如今处于搜索、协助和代理工作流的关键路径中,语义缓存对于降低推理成本和延迟至关重要。生产部署通常采用分层静态-动态设计:静态缓存由经过筛选的离线验证响应组成,由动态缓存补充。在实践中,两个层级通常由单一嵌入相似度阈值控制,导致硬性权衡:保守阈值会错过安全重用机会,而激进阈值则可能提供语义不正确的响应。我们引入Krites,一种异步、由LLM判断的缓存策略,扩展静态覆盖范围而不改变服务决策。在关键路径上,Krites的行为与标准静态阈值策略完全相同。当提示的最近静态邻居刚好低于静态阈值时,Krites异步调用LLM法官验证静态响应是否适用于新提示。批准的匹配被提升到动态缓存,允许未来重复和改写使用经过筛选的静态答案,并随着时间推移扩展静态范围。在基于跟踪的模拟中,Krites在对话和搜索工作负载上将使用经过筛选的静态答案的请求比例(直接静态命中加验证提升)提高了3.9倍,与调优基线相比,关键路径延迟保持不变。

英文摘要:

Large language models (LLMs) now sit in the critical path of search, assistance, and agentic workflows, making semantic caching essential for reducing inference cost and latency. Production deployments typically use a tiered static-dynamic design: a static cache of curated, offline vetted responses mined from logs, backed by a dynamic cache populated online. In practice, both tiers are commonly governed by a single embedding similarity threshold, which induces a hard tradeoff: conservative thresholds miss safe reuse opportunities, while aggressive thresholds risk serving semantically incorrect responses. We introduce Krites, an asynchronous, LLM-judged caching policy that expands static coverage without changing serving decisions. On the critical path, Krites behaves exactly like a standard static threshold policy. When the nearest static neighbor of the prompt falls just below the static threshold, Krites asynchronously invokes an LLM judge to verify whether the static response is acceptable for the new prompt. Approved matches are promoted into the dynamic cache, allowing future repeats and paraphrases to reuse curated static answers and expanding static reach over time. In trace-driven simulations on conversational and search workloads, Krites increases the fraction of requests served with curated static answers (direct static hits plus verified promotions) by up to3.9 times for conversational traffic and search-style queries relative to tuned baselines, with unchanged critical path latency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑