AMV-L：用于长运行LLM系统中尾延迟控制的生命周期管理代理内存
AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems
AI总结：
AMV-L通过生命周期管理内存，有效降低LLM长运行系统的延迟和吞吐量，提升极端尾部性能。
摘要：
长运行的LLM代理需要持久内存以在交互中保持状态，但大多数部署系统使用基于年龄的保留（例如TTL）来管理内存。虽然TTL限制了项目的生命周期，但它并不限制内存在请求路径上的计算足迹：随着保留项的积累，检索候选集和向量相似性扫描可以无规律地增长，导致重尾延迟和不稳定的吞吐量。我们提出了AMV-L（自适应内存价值生命周期），一种内存管理框架，将代理内存视为受管理的系统资源。AMV-L为每个内存项分配一个持续更新的效用分数，并使用价值驱动的提升、降级和驱逐来维护生命周期层级；检索被限制在受限制的、层级感知的候选集中，将请求路径的工作集与总保留内存解耦。我们在完整的LLM服务系统中实现了AMV-L，并在相同的长运行工作负载下，与两个基线进行评估：TTL和LRU工作集策略，带有固定的提示注入上限。与TTL相比，AMV-L的吞吐量提高了3.1倍，延迟减少了4.2倍（中位数），4.7倍（p95），4.4倍（p99），同时将超过2秒的请求比例从13.8%降低到0.007%。与LRU相比，AMV-L在中位数/p95延迟上略有下降（+26% / +3%），但改进了极端尾部行为（-15% p99；-98% >2s）和更低的令牌开销（大约6%更少的令牌/请求），同时保持检索质量（价值均值在大约0-2%范围内）。这些收益主要来自限制检索集大小和向量搜索工作，而不是缩短提示。我们的结果表明，长运行LLM代理的可预测性能需要显式控制内存工作集大小和价值驱动的生命周期管理，而不是仅依赖保留时间。
英文摘要：
Long-running LLM agents require persistent memory to preserve state across interactions, yet most deployed systems manage memory with age-based retention (e.g., TTL). While TTL bounds item lifetime, it does not bound the computational footprint of memory on the request path: as retained items accumulate, retrieval candidate sets and vector similarity scans can grow unpredictably, yielding heavy-tailed latency and unstable throughput. We present AMV-L (Adaptive Memory Value Lifecycle), a memory-management framework that treats agent memory as a managed systems resource. AMV-L assigns each memory item a continuously updated utility score and uses value-driven promotion, demotion, and eviction to maintain lifecycle tiers; retrieval is restricted to a bounded, tier-aware candidate set that decouples the request-path working set from total retained memory. We implement AMV-L in a full-stack LLM serving system and evaluate it under identical long-running workloads against two baselines: TTL and an LRU working-set policy, with fixed prompt-injection caps. Relative to TTL, AMV-L improves throughput by 3.1x and reduces latency by 4.2x (median), 4.7x (p95), and 4.4x (p99), while reducing the fraction of requests exceeding 2s from 13.8% to 0.007%. Compared to LRU, AMV-L trades a small regression in median/p95 latency (+26% / +3%) for improved extreme-tail behavior (-15% p99; -98% >2s) and lower token overhead (approximately 6% fewer tokens/request), while matching retrieval quality (value means within approximately 0-2%). The gains arise primarily from bounding retrieval-set size and vector-search work, not from shortening prompts. Our results show that predictable performance for long-running LLM agents requires explicit control of memory working-set size and value-driven lifecycle management, rather than retention time alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
