检索增强生成通过整合外部知识增强了大型语言模型,这减少了幻觉但增加了提示长度。这种增加导致计算成本更高且时间到第一个令牌(Time to First Token,TTFT)更长。为缓解这一问题,现有解决方案旨在重用每个检索块的预处理KV缓存以加速RAG。然而,缺乏跨块的上下文信息导致生成质量显著下降,使KV缓存重用的潜在好处未能得到充分实现。挑战在于如何在重用块的预计算KV缓存的同时保持生成质量。我们提出了FusionRAG,一种新的推理框架,优化了RAG的预处理和重处理阶段。在离线预处理阶段,我们将其他相关文本块的信息嵌入到每个块中,而在在线重处理阶段,我们重新计算模型关注的令牌的KV缓存。结果,我们实现了生成质量和效率之间的更好权衡。根据我们的实验,与先前最先进的解决方案相比,FusionRAG在相同重新计算比率下显著提高了生成质量。通过重新计算少于15%的令牌,FusionRAG在基准上实现了高达70%的归一化F1分数,并将TTFT减少了2.66倍至9.39倍。
英文摘要:
Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token (TTFT). To mitigate this issue, existing solutions aim to reuse the preprocessed KV cache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KV cache reuse largely unfulfilled. The challenge lies in how to reuse the precomputed KV cache of chunks while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KV cache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized F1 scores than baselines and reduces TTFT by 2.66x-9.39x compared to Full Attention.
DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference
Westlake University(西湖大学);Zhejiang University(浙江大学);Carnegie Mellon University(卡内基梅隆大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Hong Kong University of Science and Technology(香港科技大学)
DeFT提出了一种基于Flash树注意力的硬件高效解码算法,通过KV-Guided Grouping和Flattened Tree KV Splitting优化树结构LLM推理中的KV缓存分区与负载均衡,显著减少IO冗余并提升GPU利用率,实现最高2.23倍端到端加速。
CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery
School of Mathematical Sciences, Beijing University of Posts and Telecommunications(北京邮电大学数学科学学院);School of Mathematics and Statistics, Chongqing University(重庆大学数学与统计学院);School of Science, Beijing Forestry University(北京林业大学理学院)
QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
Zhejiang University(浙江大学);East China Normal University(华东师范大学);Ant Group(蚂蚁集团);The Hong Kong Polytechnic University(香港理工大学);Zhejiang Normal University(浙江师范大学);Tongji University(同济大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学)
Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs
Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信(TeleAI)人工智能研究所);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Individual Researcher(独立研究者)
School of Information, Renmin University of China(中国人民大学信息学院);Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China(数据工程与知识工程重点实验室);Engineering Research Center of Database and Business Intelligence, Beijing, China(数据库与商业智能工程研究中心);Ant Group, Hangzhou, China(蚂蚁集团)
Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents
Carnegie Mellon University(卡内基梅隆大学);Capital One(第一资本);University of Chicago(芝加哥大学);University of California, Berkeley(加州大学伯克利分校);University of Southern California(南加州大学);University of Washington(华盛顿大学)
TokenPilot: Cache-Efficient Context Management for LLM Agents
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Xi’an University of Electronic Science and Technology(西安电子科技大学);HomologyAI(同源人工智能)