前缀缓存语言模型服务中的精确内存-时间优化
Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
浏览论文内容
中文总结 AI 辅助
针对前缀缓存语言模型服务，提出前缀证书保留（PCR）精确优化模型，通过最小割和断点定理求解，并在Mooncake数据集上验证，实现内存-时间权衡的优化。
中文摘要 AI 辅助
保留语言模型的前缀状态需要在重计算与存储时间之间进行权衡。独立优化每个缓存块可能会高估节省量：一个驻留块仅在所需的前面前缀也可用时才可用。我们引入了前缀证书保留（PCR），一种针对静态、分组、重置访问超时的精确有限轨迹公式。可用前缀奖励成为节点，其前提是超时阈值和前面的命中证书。由此产生的最大权重闭包可简化为一次最小割，图大小与块查找次数和超时选择数量呈线性关系。一个断点定理将该构造扩展到所有非负超时，且无离散化误差。我们还推导了一个网格大小上的线性时间动态规划，用于有序超时，并给出了证明该限制成本的界限。对39,632个公开Mooncake请求的详尽小实例检查和按时间顺序重放验证了该公式。在固定网格上，有序超时在120个轨迹-分组-价格案例中的118个达到了无约束训练最优。异构保留改善了几种留出内存-时间权衡，但更精细的训练优化并未均匀地改善迁移。本贡献是一个易处理的优化模型和用于保留策略的可审计基准；实验测量的是可用前缀块和存储时间，而非GPU延迟。
英文摘要
Retaining language-model prefix states trades recomputation against storage time. Optimizing each cached block independently can overcount savings: a resident block is usable only when the required preceding prefix is also available. We introduce Prefix-Certificate Retention (PCR), an exact finite-trace formulation for static, grouped, reset-on-access timeouts. Usable-prefix rewards become nodes whose prerequisites are timeout thresholds and preceding hit certificates. The resulting maximum-weight closure reduces to one minimum cut, with graph size linear in the number of block lookups and timeout choices. A breakpoint theorem extends the construction to all nonnegative timeouts without discretization error. We also derive a linear-time-in-grid-size dynamic program for ordered timeouts and bounds that certify the cost of this restriction. Exhaustive small-instance checks and chronological replay of 39,632 public Mooncake requests validate the formulation. On the fixed grid, ordered timeouts attain the unrestricted training optimum in 118 of 120 trace-grouping-price cases. Heterogeneous retention improves several held-out memory-time tradeoffs, but finer training optimization does not uniformly improve transfer. The contribution is a tractable optimization model and an auditable benchmark for retention policies; the experiments measure usable prefix blocks and storage time, not GPU latency.
发表机构
- Independent Researcher（独立研究者）
机构由 AI 辅助整理，请以论文原文为准。
