KV缓存修复预算应投向何处?非前缀复用中的干扰项结构与上下文兼容性
Where Should a KV-Cache Repair Budget Go? Distractor Structure and Contextual Compatibility in Non-Prefix Reuse
AI总结:
该研究针对非前缀KV缓存复用的修复预算分配问题,通过实验发现应将预算投向目标片段,重新计算目标的方法在多数场景优于插入完整预填充KV,且在部分任务中优于InfoFlow式重排序。
摘要:
非前缀KV缓存复用会对检索到的片段进行独立编码并拼接其缓存。为确定少量修复预算应投向何处,我们在3个7B-8B模型和4个对话记忆、开放域问答、API文档数据集上,改变了包含答案的目标片段周围的其他片段。复用会丢失完整预填充可回答的19%-100%依赖上下文的问题,且当其他片段与目标属于同一类型时,丢失情况更严重。由于可恢复的损失分布在这些其他片段上,仅恢复其中少数几乎无效果,因此预算应投向目标。这一方法通过上下文兼容性实现:在所有7个测试场景中,针对复用缓存重新计算目标,优于插入其完整预填充KV,尽管重新计算的KV与完整预填充KV仍存在较大差距。不过,复用缓存上的问题注意力最多仅在18.8%的开发问题中选中目标。检索器能更频繁找到目标,在9个开放域问答和API文档场景中的6个里,将目标移至问题旁后重新计算其顶部片段,在相同5%预算下显著优于InfoFlow式重排序。但在对话记忆任务中,重新计算片段从未显著优于各场景所选的令牌级选择器。
英文摘要:
Non-prefix KV-cache reuse encodes retrieved chunks independently and concatenates their caches. To find where a small repair budget should be spent, we vary the chunks around the answer-bearing one (the target) across three 7B-8B models and four datasets of dialogue memory, open-domain QA and API documentation. Reuse loses 19-100% of the context-dependent questions that full prefill answers, and it loses more when the other chunks are of the same kind as the target. Because the recoverable damage is spread over these other chunks, restoring a few of them recovers little, and the budget is better spent on the target. This works through contextual compatibility: recomputing the target against the reused cache beats inserting its full-prefill KV in all seven settings tested, although the recomputed KV stays far from the full-prefill KV. Question attention over reused caches, however, selects the target in at most 18.8% of development questions. A retriever finds it more often, and recomputing its top chunk after moving it next to the question significantly beats InfoFlow-style reordering at the same 5% budget in six of nine open-domain QA and API-documentation settings. On dialogue memory, though, recomputing chunks never significantly beats the token-level selector chosen for each setting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。