StepCache: 基于轻量验证和选择性修补的步骤级重用用于大语言模型服务
StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving
AI总结：
StepCache通过步骤级重用和轻量验证实现LLM服务的高效处理，减少延迟并提高准确性，支持结构化输出和容错机制。
摘要：
我们针对LLM服务工作负载进行了研究，其中重复请求共享相同的解决方案结构但局部约束不同，如输出模式、变量名或数值常量。传统缓存方法通常重用完整响应（语义缓存）或模型内部的KV/前缀状态，这些方法在部分变化下容易破裂或紧密耦合到特定后端。我们提出了StepCache，一种与后端无关的步骤级重用层，将输出分割为有序步骤，检索最佳匹配的缓存请求，使用轻量级任务感知检查验证步骤，并通过选择性修补仅重生成失败区域。StepCache还支持严格的结构化输出强制执行，包括单步提取、必需键约束和一次性修复，以及保守的跳过重用回退用于语义变化。对于线性方程，StepCache通过有界修复循环促进验证到修正，具有确定性回退，当后端模型失败时保证正确性。在CPU-only扰动密集的微基准测试中，针对数学和JSON变体，平均延迟从2.13秒降至0.67秒，中位数延迟从2.42秒降至0.01秒，p95延迟从3.38秒降至3.30秒。它还减少了总token使用量从36.1k到27.3k，并在任务特定检查和拼接输出完整性检查下将端到端正确性从72.5%提高到100%。在请求中，79.7%采用仅重用的快速路径，5.4%需要修补，14.9%触发跳过重用。
英文摘要：
We address LLM serving workloads where repeated requests share a common solution structure but differ in localized constraints, such as output schema, variable names, or numeric constants. Prior caching approaches typically reuse either full responses (semantic caching) or model-internal KV/prefix states, which are respectively brittle under partial changes or tightly coupled to specific backends. We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps, retrieves the best-matching cached request, verifies steps using lightweight task-aware checks, and regenerates only failing regions via selective patching. StepCache additionally supports strict structured-output enforcement for JSON, including single-step extraction, required-key constraints, and one-shot repair, as well as conservative skip-reuse fallbacks for semantic changes. For linear equations, StepCache promotes verification into correction via a bounded repair loop with a deterministic fallback that guarantees correctness when the backend model fails. In a CPU-only perturbation-heavy micro-benchmark on math and JSON variants, averaged over three seeds, StepCache reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s. It also reduces total token usage from 36.1k to 27.3k and improves end-to-end correctness from 72.5% to 100% under task-specific checks and a stitched-output integrity check. Across requests, 79.7% take the reuse-only fast path, 5.4% require patching, and 14.9% trigger skip-reuse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
