arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.03475cs.LG
收藏

ContextPilot: 通过上下文重用实现快速长上下文推断

ContextPilot: Fast Long-Context Inference via Context Reuse

Yinsicheng Jiang, Yeqi Huang, Liang Cheng, Cheng Deng, Xuan Sun, Luo Mai

AI总结:

ContextPilot通过引入上下文重用机制,显著提升长上下文推断效率,减少延迟并保持推理质量。

摘要:

人工智能应用日益依赖长上下文推断,其中大型语言模型(LLMs)消耗大量上下文以支持更强的推理能力。常见例子包括检索增强生成、代理记忆层和多代理协调。随着输入上下文变长,prefill延迟成为主要瓶颈。然而,目前的prefill加速技术面临权衡:要么保持推理质量但提供很少的KV缓存重用,要么提高重用率却导致推理质量下降。我们提出了ContextPilot系统,通过引入上下文重用作为新的机制来加速长上下文推断。ContextPilot引入了一个上下文索引,用于识别LLM交互中的重叠上下文块(例如跨用户和回合)。它进一步提出了上下文排序和去重技术,以最大化KV缓存重用。为了在重用过程中保持推理质量,它引入了简洁的上下文注释,以防止质量下降。最后,ContextPilot围绕一个模块化架构构建,具有清晰的接口,能够与现有推理引擎集成。广泛的评估表明,ContextPilot将LLM的prefill延迟减少了高达3倍,相比最先进的方法,同时保持推理质量。在更长的上下文长度下,它甚至可以提高推理质量。ContextPilot在:https://github.com/EfficientContext/ContextPilot上开源。

英文摘要:

AI applications increasingly depend on long-context inference, where LLMs consume substantial context to support stronger reasoning. Common examples include retrieval-augmented generation, agent memory layers, and multi-agent orchestration. As input contexts get longer, prefill latency becomes the main bottleneck. Yet today's prefill acceleration techniques face a trade-off: they either preserve reasoning quality but deliver little KV-cache reuse, or improve reuse at the cost of degraded reasoning quality. We present ContextPilot, a system that accelerates prefill by introducing context reuse as a new mechanism for faster long-context inference. ContextPilot introduces a context index to identify overlapping context blocks across LLM interactions (e.g., across users and turns). It further proposes context ordering and de-duplication techniques to maximize KV-cache reuse. To preserve reasoning quality under reuse, it introduces succinct context annotations that prevent quality degradation. Finally, ContextPilot is built around a modular architecture with a clean interface that integrates with existing inference engines. Extensive evaluation shows that ContextPilot reduces LLM prefill latency by up to $3\times{}$ compared to state-of-the-art methods while preserving reasoning quality. At longer context lengths, it can even improve reasoning quality. ContextPilot is open-sourced at: https://github.com/EfficientContext/ContextPilot.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑