面向人类审批 timescale 的大语言模型智能体的自适应 KV 保留机制

中文摘要 AI 辅助

现有智能体服务系统针对的是秒级的工具调用停顿，而智能体大语言模型（LLM）请求在等待人类审批时可能会被暂停数分钟甚至数小时。我们研究了暂停与恢复对 GPU 服务性能的影响，并开发了一种保留策略，用于在不确定的审批等待时间下平衡活跃服务容量与未来重新计算的成本。核心矛盾十分突出：保留暂停的 KV 能实现快速恢复，但会消耗足够多的 GPU 容量，导致活跃服务的有效吞吐量（goodput）降低 41%；而驱逐暂停的 KV 可避免该驻留成本，但当请求返回时，恢复延迟会高出近 10 倍。我们围绕 GPU 机会成本开发了分层保留控制器，该控制器以通用的 GPU 时间成本来表示保留或重建暂停请求的 KV 状态所消耗的服务容量。在主机内存中，控制器利用校准后的等待样本在无限期保留与按加载索引的过期策略之间进行选择，无需对每个请求的等待时间进行预测。在人类规模的审批工作负载上，与 vLLM 基准相比，我们的控制器将活跃请求的有效吞吐量提升了 23% 至 51%；与 MORI 相比提升了 22% 至 29%；与 Continuum 相比提升了 41% 至 52%。

英文摘要

Unlike the seconds-scale tool-call pauses targeted by prior agent-serving systems, agentic LLM requests can be suspended for minutes or hours while waiting for human approval. We study how suspension and resumption affect GPU serving performance and develop a retention policy that balances active-serving capacity against future recomputation under uncertain approval waits. The central tension is severe because retaining suspended KV preserves fast resume but can consume enough GPU capacity to reduce active-serving goodput by 41%, while evicting it avoids that residency cost at the expense of nearly $10\times$ higher resume latency when the request returns. We develop a tiered retention controller around GPU opportunity cost, which expresses the serving capacity consumed by preserving or reconstructing a suspended request's KV state in a common GPU-time cost. Within host memory, the controller selects between indefinite retention and load-indexed expiration using calibration wait samples, without requiring per-request wait prediction. On human-scale approval workloads, our controller improves active-request goodput by 23-51% over the vLLM baselines, 22-29% over MORI, and 41-52% over Continuum.

发表机构

• Johns Hopkins University（约翰斯·霍普金斯大学）

机构由 AI 辅助整理，请以论文原文为准。