arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12108cs.AI
收藏

Pensieve范式:具备状态的语言模型掌握自身上下文

The Pensieve Paradigm: Stateful Language Models Mastering Their Own Context

Tencent AI Lab(腾讯人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Xiaoyuan Liu, Tian Liang, Dongyang Ma, Deyu Zhou, Haitao Mi, Pinjia He, Yan Wang

AI总结:

StateLM 通过内部推理循环和记忆工具,使语言模型具备状态管理能力,从而在长文档问答、聊天记忆和深度研究任务中取得显著性能提升。

摘要:

在《哈利波特》的世界中,当邓布利多的记忆负担过重时,他会将记忆提取到 Pensieve 中以供日后回顾。在人工智能的世界中,尽管我们拥有 Pensieve 成熟的数据库和检索系统,但我们的模型却缺乏操作它的'魔杖'。它们仍然像没有自主能力的邓布利多,被动地接受人工工程化的上下文作为整个记忆。本文最终将魔杖交到了模型手中。我们引入了 StateLM,一种新的基础模型,配备了内部推理循环以管理自身状态。我们为模型配备了各种记忆工具,如上下文修剪、文档索引和笔记记录,并训练它主动管理这些工具。通过学习动态工程自己的上下文,我们的模型摆脱了固定窗口的建筑牢笼。在各种模型规模上的实验表明,StateLM 在多样化的场景中均表现出色。在长文档问答任务中,StateLM 在所有模型规模上均优于标准 LLM;在聊天记忆任务中,它们在标准 LLM 上实现了 10% 到 20% 的绝对准确率提升。在深度研究任务 BrowseComp-Plus 上,性能差距更加明显:StateLM 达到 52% 的准确率,而标准 LLM 对手则在 5% 左右挣扎。最终,我们的方法使 LLM 从被动预测者转变为具有状态的智能体,其中推理成为一种具有状态且可管理的过程。

英文摘要:

In the world of Harry Potter, when Dumbledore's mind is overburdened, he extracts memories into a Pensieve to be revisited later. In the world of AI, while we possess the Pensieve-mature databases and retrieval systems, our models inexplicably lack the "wand" to operate it. They remain like a Dumbledore without agency, passively accepting a manually engineered context as their entire memory. This work finally places the wand in the model's hand. We introduce StateLM, a new class of foundation models endowed with an internal reasoning loop to manage their own state. We equip our model with a suite of memory tools, such as context pruning, document indexing, and note-taking, and train it to actively manage these tools. By learning to dynamically engineering its own context, our model breaks free from the architectural prison of a fixed window. Experiments across various model sizes demonstrate StateLM's effectiveness across diverse scenarios. On long-document QA tasks, StateLMs consistently outperform standard LLMs across all model scales; on the chat memory task, they achieve absolute accuracy improvements of 10% to 20% over standard LLMs. On the deep research task BrowseComp-Plus, the performance gap becomes even more pronounced: StateLM achieves up to 52% accuracy, whereas standard LLM counterparts struggle around 5%. Ultimately, our approach shifts LLMs from passive predictors to state-aware agents where reasoning becomes a stateful and manageable process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑