arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.17686cs.CLcs.LG
收藏

将上下文编码为大语言模型权重:什么构成了上下文?

Encoding Context into LLM Weights: What Makes It Context?

The University of Melbourne(墨尔本大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);The University of Tokyo(东京大学);Hong Kong Baptist University(香港浸会大学)

Yue Wang, Qizhou Wang, Feng Liu, Gang Niu, Masashi Sugiyama, Bo Han

AI总结:

本研究针对LLM上下文编码的推理成本问题,提出关联标识符与提示框架的方法,使编码信息能被正确用作上下文,在多任务及基准测试中优于仅用现成LLM的基线。

摘要:

更长的上下文使大语言模型(LLMs)能够基于更多信息调整预测,但推理成本随上下文长度急剧增长,令人难以承受。一种新兴替代方案采用测试时更新,将上下文编码为模型权重的临时变化。然而,将此信息编码到已代表通用知识的权重中,会模糊其作为上下文的作用,导致LLM可能无法正确利用它来调整后续推理。为恢复此上下文作用,我们在编码期间为每个上下文关联一个标识符,从而将新编码的信息与预先存在的知识区分开;在推理时调用该标识符,使LLM能够复述编码的上下文。不过,成功复述并不确保LLM会将回忆的信息用作上下文,因此我们进一步引入提示框架,明确引导LLM将此信息用作上下文。在多种受控任务和实际基准中,我们的方法优于仅使用现成LLM的基线,展现出在上下文编码方面的优势。

英文摘要:

Longer contexts allow large language models (LLMs) to condition their predictions on more information, but inference costs grow prohibitively with context length. An emerging alternative uses test-time updates to encode context as temporary changes to model weights. However, encoding this information into weights that already represent general knowledge obscures its role as context, so the LLM may fail to use it properly to condition subsequent inference. To restore this contextual role, we first associate each context with an identifier during encoding, thereby distinguishing the newly encoded information from pre-existing knowledge. Invoking the identifier at inference then enables the LLM to recite the encoded context. Nevertheless, successful recitation does not ensure that the LLM will use the recalled information as context. We therefore further introduce a prompting framework that explicitly guides the LLM to use this information as context. Across diverse controlled tasks and practical benchmarks, our method outperformed the baseline using only off-the-shelf LLMs, showing its advances in context encoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑