arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09494cs.CLcs.AIcs.IRcs.LG
收藏

RecaLLM:通过显式上下文检索解决‘思维迷失’现象

RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Kyle Whitecross, Negin Rahimi

AI总结:

RecaLLM通过交替推理与显式上下文检索解决推理过程中因长上下文导致的检索性能下降问题,显著提升了RULER和HELMET基准测试表现。

摘要:

我们提出RecaLLM,一种经过后训练的推理语言模型,旨在有效利用长上下文信息。上下文检索通过识别相关证据支持推理,而推理又影响检索内容。然而,两者交互仍不明确。在多个开源LLM的初步实验中,我们发现即使在短推理跨度后,上下文检索性能显著下降,揭示了测试时扩展的关键瓶颈,即‘思维迷失’:提升性能的推理步骤反而使后续检索更困难。为解决此问题,RecaLLM交替推理与显式上下文检索,解决子问题所需上下文信息。我们引入一种轻量级约束解码机制,使证据片段可直接复制,增强后续生成的依据。在多样化的词汇和语义检索任务上训练,RecaLLM在RULER和HELMET两个长上下文基准测试中表现优异,显著优于基线。值得注意的是,使用最多10K token训练样本,在128K token上下文窗口中观察到一致收益,远低于现有长上下文方法,展示了改进长上下文性能的有希望路径。

英文摘要:

We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑