预计算记忆的组合、重建与修正的成本分析

中文摘要 AI 辅助

语言模型可从预计算记忆中获取答案，预计算记忆是模型对某一材料主体的已保存读取结果，可在不同请求间复用，无需每次重新读取。本文明确了该做法在哪些情况下能保持正确性，以及在何种条件下会失效。针对 Llama-3.1-8B-Instruct 开展的实验中，同时使用了已保存的键值缓存及其经训练的压缩形式，结果显示：预计算记忆若由分别准备的部分组合而成会出现性能下降；在本研究的测量中，预计算记忆要保持时效性，需进行重建，而重建成本占完整准备成本的很大比例；此外，预计算记忆会忽略与其一同提供的、基于措辞条件的修正。若预计算记忆能相互提供、可经济高效地重建，且能被实时到达的新信息替代，那么它们可作为一种避免在重复查询中向模型重新输入上下文的方式。本研究结果对处理各类查询的部署系统的启示是：预计算记忆最好按照新信息改变其原始计算依据的频率进行重建。对键值缓存的训练压缩形式进行热重建，以及在记忆旁提供特定措辞的更新（如粘贴文本或注入缓存状态），这两种方式在保持预计算记忆时效性方面展现出特别前景，后者可作为两次重建之间的临时措施，本研究还测量了每种方式的成本，并指出了与之相关的剩余问题。

英文摘要

Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at each. This paper maps where that practice preserves correctness and the conditions under which it fails. Across experiments on Llama-3.1-8B-Instruct using both saved key-value caches and trained compressions of them, precomputed memory degrades when assembled from separately prepared parts, stays current only through rebuilds costing a large fraction of full preparation in our measurements, and ignores corrections served beside it conditional on phrasing. If precomputed memories can be served alongside one another, be cost-efficiently rebuilt, and be superseded by new information arriving in real-time, they can serve as a way to avoid re-feeding context to a model over repeated queries. The implication of our results for a deployed system that deals with a variety of queries is that precomputed memories are best rebuilt on the cadence at which new information changes what the memory was originally computed from. Both warm-rebuilding trained compressions of key-value caches and serving specifically-phrased updates beside a memory, as pasted text or injected cache state, show particular promise for keeping precomputed memories current, the latter as an interim measure between rebuilds, and we measure the cost and name the remaining questions associated with each.

发表机构

• Williams College（威廉姆斯学院）

机构由 AI 辅助整理，请以论文原文为准。