arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00151cs.DC
收藏

SmoothAgent: 基于前瞻上下文工程的高效长程LLM Agent服务

SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering

University of California, San Diego(加利福尼亚大学圣地亚哥分校)

Zaifeng Pan, Qianxu Wang, Zhengding Hu, Chang Chen, Yue Guan, Yanbo Zhou, Steven Swanson, Yufei Ding

AI总结:

针对LLM Agent中上下文变换导致KV缓存失效和TTFT增加的问题,提出前瞻编程模型和调度器,实现异步上下文预处理,降低TTFT达11.9倍。

摘要:

基于LLM的Agent执行多轮工作流,上下文不断增长,LLM调用与工具调用和环境反馈交替进行。为了保持模型质量,现代Agent框架依赖于上下文工程策略,如卸载、缩减和隔离来控制上下文长度。然而,这些策略引入了显著的上下文变换开销:每次变换会使现有的KV缓存失效并触发重新预填充,导致首个令牌生成时间(TTFT)增加。在本文中,我们识别出上下文变换是段可分解的,即前缀的变换独立于未来的令牌。这一性质使得变换可以提前执行。基于这一洞察,我们提出了一种前瞻编程模型,允许Agent框架将上下文变换表示为异步操作,而无需修改其执行逻辑。运行时主动执行这些变换并提前准备变换后的KV缓存,从而实现直接的上下文替换而不阻塞。我们进一步在LLM服务系统中设计了一个前瞻感知调度器,以支持这些异步请求与延迟关键型工作负载一起运行,并控制干扰。我们实现了我们的方法以支持代表性的上下文工程策略,并将其集成到现有的Agent框架和LLM服务系统中。实验表明,我们的方法有效消除了变换开销,并将TTFT降低了高达11.9倍。

英文摘要:

LLM-based agents execute multi-turn workflows with continuously growing contexts, where LLM calls are interleaved with tool invocations and environment feedback. To maintain model quality, modern agent frameworks rely on context engineering strategies such as offloading, reduction, and isolation to control the context length. However, these strategies introduce significant context transformation overhead: each transformation invalidates existing KV caches and triggers re-prefill, leading to increased time-to-first-token (TTFT). In this paper, we identify that context transformations are segment-decomposable, where the transformation of a prefix is independent of future tokens. This property enables transformations to be executed ahead of time. Based on this insight, we propose a lookahead programming model that allows agent frameworks to express context transformations as asynchronous operations without modifying their execution logic. The runtime proactively executes these transformations and prepares transformed KV caches in advance, enabling direct context replacement without blocking. We further design a lookahead-aware scheduler in LLM serving systems to support these asynchronous requests alongside latency-critical workloads with controlled interference. We implement our approach to support representative context engineering strategies and integrate it into existing agent frameworks and LLM serving systems. Experiments show that our approach effectively eliminates transformation overhead and reduces TTFT by up to 11.9x.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑