arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.24601cs.AIcs.CL
收藏

递归语言模型

Recursive Language Models

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)

Alex L. Zhang, Tim Kraska, Omar Khattab

AI总结:

本文提出递归语言模型(RLMs),通过将长提示视为外部环境的一部分,使大语言模型能够递归调用自身处理长输入,从而在多个长上下文任务中显著提升性能。

摘要:

我们研究了允许大语言模型(LLMs)通过推理时间扩展处理任意长提示的方法。我们提出了递归语言模型(RLMs),一种通用的推理范式,将长提示视为外部环境的一部分,允许LLM程序性地检查、分解并递归调用自身处理提示片段。我们发现RLMs可以成功处理比模型上下文窗口大两个数量级的输入,并且即使对于较短的提示,也大幅优于常规前沿LLM和常见长上下文及编码支架(例如在GPT-5上,相对于压缩,中位数为26%,相对于CodeAct带子调用,为130%,相对于Claude Code为13%)在四个多样化的长上下文任务中,同时具有可比的成本。在小规模上,我们对第一个RLM模型进行了后训练。我们的模型RLM-Qwen3-8B在平均上比底层Qwen3-8B模型提高了28.3%,甚至在三个长上下文任务中接近常规GPT-5的性能。代码可在https://github.com/alexzhang13/rlm上获得。

英文摘要:

We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive Language Models (RLMs), a general inference paradigm that treats long prompts as part of an external environment and allows the LLM to programmatically examine, decompose, and recursively call itself over snippets of the prompt. We find that RLMs can successfully process inputs up to two orders of magnitude beyond model context windows and, even for shorter prompts, dramatically outperform the quality of vanilla frontier LLMs and common long-context and coding scaffolds (e.g., on GPT-5 by a median across the evaluated benchmarks of $26\%$ against compaction, $130\%$ against CodeAct with sub-calls, and $13\%$ against Claude Code) across four diverse long-context tasks while having comparable cost. At a small scale, we post-train the first model around the RLM. Our model, RLM-Qwen3-8B, outperforms the underlying Qwen3-8B model by $28.3\%$ on average and even approaches the quality of vanilla GPT-5 on three long-context tasks. Code is available at https://github.com/alexzhang13/rlm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑