arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.12904cs.CLcs.AI
收藏

从前缀缓存到融合RAG缓存:加速检索增强生成中的LLM推理

From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation

Hangzhou Dianzi University(杭州电子科技大学);Tsinghua University(清华大学)

Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxing Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jinqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu, Congfeng Jiang

AI总结:

FusionRAG通过优化RAG的预处理和重处理阶段,在保持生成质量的同时提升推理效率。

摘要:

检索增强生成通过整合外部知识增强了大型语言模型,这减少了幻觉但增加了提示长度。这种增加导致计算成本更高且时间到第一个令牌(Time to First Token,TTFT)更长。为缓解这一问题,现有解决方案旨在重用每个检索块的预处理KV缓存以加速RAG。然而,缺乏跨块的上下文信息导致生成质量显著下降,使KV缓存重用的潜在好处未能得到充分实现。挑战在于如何在重用块的预计算KV缓存的同时保持生成质量。我们提出了FusionRAG,一种新的推理框架,优化了RAG的预处理和重处理阶段。在离线预处理阶段,我们将其他相关文本块的信息嵌入到每个块中,而在在线重处理阶段,我们重新计算模型关注的令牌的KV缓存。结果,我们实现了生成质量和效率之间的更好权衡。根据我们的实验,与先前最先进的解决方案相比,FusionRAG在相同重新计算比率下显著提高了生成质量。通过重新计算少于15%的令牌,FusionRAG在基准上实现了高达70%的归一化F1分数,并将TTFT减少了2.66倍至9.39倍。

英文摘要:

Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token (TTFT). To mitigate this issue, existing solutions aim to reuse the preprocessed KV cache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KV cache reuse largely unfulfilled. The challenge lies in how to reuse the precomputed KV cache of chunks while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KV cache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized F1 scores than baselines and reduces TTFT by 2.66x-9.39x compared to Full Attention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑