arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.40545cs.AI
收藏

检索即遍历:面向有状态长上下文智能体的值索引记忆

Retrieval as Traversal: Value-Indexed Memory for Stateful Long-Context Agents

Technical University of Munich(慕尼黑工业大学);Nanjing University(南京大学)

Daniel Strauß, Zhenshan Bing, Alois Knoll

AI总结:

本研究针对长上下文智能体的检索问题,提出值索引记忆方法,通过局部遍历地址空间分离记忆内容与地址,在MiniGrid Memory实验中其训练效率优于高斯与分类绝对寻址。

摘要:

长上下文模型会反复解决高度相关的检索问题。注意力机制或全局检索可存储大量信息,但每次前向传播仍需搜索关键内容;循环压缩仅通过提前确定要保留的信息来避免这种搜索,这两种机制均未直接利用相邻检索步骤中有用记忆可能存在的大量重叠。我们提出一种替代方案,将检索定义为通过地址空间的持续局部遍历,将记忆的内容与地址分离:效用由检索内容所揭示的信息定义,几何结构仅决定该信息的可达成本,这形成了成本正则化的索引目标——在局部邻域内保留信息价值,同时对每个本地可访问的记忆收费,其边际价值形式兼具吸引与排斥压力:当互补记忆的附加价值超过检索成本时,应可共同检索;而冗余记忆不应仅因各自有用就占据同一检索集。作为原理验证,我们仅在MiniGrid Memory上测试了最简单的固定几何实例:持续局部遍历头在5个token的上下文下,经0.59M训练步骤后达到0.95的平均回报,而高斯绝对寻址需0.94M,分类绝对寻址则超过2M。该实验未测试学习几何结构或重复检索,反而提出假设:有状态检索可能比无状态检索更易学习,需进一步实验验证完整的所提方法。

英文摘要:

Long-context models repeatedly solve closely related retrieval problems. Attention or global retrieval can retain large stores, but each forward pass again searches for what matters. Recurrent compression avoids this search only by deciding earlier what information to preserve. Neither mechanism directly exploits that useful memories at neighbouring retrieval steps may substantially overlap. We propose an alternative in which retrieval is a persistent local traversal through an address space. We separate a memory's content from its address. Utility is defined by the information revealed by retrieving the content, while geometry determines only how cheaply that information can be reached. This leads to a cost-regularised indexing objective that preserves information value inside local neighbourhoods while charging for every memory made locally accessible. Its marginal-value form gives both attractive and exclusion pressures. Thus, complementary memories should be co-retrievable when the additional value exceeds retrieval cost, whereas redundant memories should not occupy the same retrieval set solely because they are individually useful. As a proof of principle, we test only the simplest fixed-geometry instance on MiniGrid Memory. A persistent local traversal head reaches $0.95$ mean return after $0.59$M training steps with a five-token context, compared with $0.94$M for Gaussian absolute addressing and more than $2$M for categorical absolute addressing. The experiment does not test learned geometry or repeated retrieval. Instead, it motivates the hypothesis that stateful retrieval may be easier to learn than stateless. To test the full proposed method further experimentation is necessary.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑