文档应驻留何处：上下文、表示还是参数？

中文摘要 AI 辅助

为了回答预训练数据之外的问题，大型语言模型（LLMs）需要获取新信息，这些信息可以以文档形式呈现在上下文窗口中，编码到模型参数中，或作为潜在表示注入。然而，每种方法在效率、成本和性能上都有不同的权衡，没有单一的最优方案。我们在五个知识密集型基准上，对基于表示（基于KV缓存）和基于参数（基于微调）的适应方法进行了受控比较。我们表明，在oracle设置中，Cartridges（KV）在几乎每个存储预算下都是最准确的注入方法，比参数化方法高出10个百分点。Compaction（KV）仅在低压缩率下与Cartridges匹配，在压缩率高于50倍时落后参数化方法10个百分点。在更现实的多文档检索场景中，Cartridges是唯一能与上下文学习（ICL）相匹配的方法，领先参数化方法29个百分点，领先Compaction 15个百分点。尽管如此，Cartridges也是除全量微调和大型MLP适配器之外，唯一遭受灾难性遗忘的方法，即在控制基准上性能下降6%，在编码任务上下降13%。

英文摘要

To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.

发表机构

• Amazon AGI（亚马逊AGI）

机构由 AI 辅助整理，请以论文原文为准。