迈向大型语言模型的上下文参数化演化
Towards Evolving Context Parameterization for Large Language Models
浏览论文内容
中文总结 AI 辅助
针对上下文参数化在动态更新中的失效问题，提出PLUME方法，通过全局与局部参数视图自适应整合，在MUSE-bench上显著提升更新整合与信息保留性能。
中文摘要 AI 辅助
上下文参数化使大型语言模型（LLMs）能够将上下文内化为可复用的模型参数，避免在后续查询中重复处理。然而，现有方法通常假设上下文是静态的，并且缺乏在持续更新下区分有效性状态的显式机制。为了研究这一现实场景，我们形式化了具有顺序演化的记忆更新（MUSE）任务，并构建了MUSE-bench基准，以评估更新整合和未受影响信息的保留。由此产生的挑战要求在调整记忆证据贡献的同时保持全局状态。受此启发，我们提出了PLUME，一种无需训练的方法，它构建全局更新表示，激活记忆证据以形成局部参数视图，并在解码过程中自适应地整合它们的预测。在MUSE-bench上的综合评估证明了PLUME在顺序演化设置中的有效性，平均ROUGE-L召回率相对提高了29.9%，LLM-as-a-Judge相对提高了54.9%。我们的代码可在以下网址获取：this https URL。
英文摘要
Context parameterization enables large language models (LLMs) to internalize contexts into reusable model parameters, avoiding repeated processing across subsequent queries. However, existing methods typically assume static contexts and lack explicit mechanisms for distinguishing validity states under continual updates. To study this real-world scenario, we formalized the Memory Updating with Sequential Evolution (MUSE) task and constructed MUSE-bench to evaluate update incorporation and unaffected-information preservation. The resulting challenge requires preserving the global state while adjusting the contribution of memory evidence. Motivated by this, we proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding. Comprehensive evaluation on MUSE-bench demonstrated PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge. Our codes are available at: https://github.com/xiaobingshi-LLM/PLUME.
发表机构
- Nanyang Technological University（南洋理工大学）
- Peking University（北京大学）
机构由 AI 辅助整理，请以论文原文为准。
