arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.09516cs.CLcs.AIcs.IR
收藏

Search-R1:通过强化学习训练大语言模型进行推理并利用搜索引擎

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

Department of Computer Science, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Center for Intelligent Information Retrieval, University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校智能信息检索中心);Google Cloud AI Research(谷歌云人工智能研究)

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, Jiawei Han

AI总结:

Search-R1通过强化学习让大语言模型在推理中自主生成搜索查询并实时检索,显著提升问答性能,在七个数据集上较RAG基线提升41%和20%。

摘要:

高效获取外部知识和最新信息对于大语言模型(LLM)的有效推理和文本生成至关重要。在推理过程中,提示具有推理能力的高级LLM使用搜索引擎往往不是最优的,因为LLM可能并未完全掌握如何与搜索引擎进行最优交互的能力。本文介绍了Search-R1,这是针对推理框架的强化学习(RL)的扩展,其中LLM在逐步推理过程中学习自主生成(多个)搜索查询并进行实时检索。Search-R1通过多轮搜索交互优化LLM的推理轨迹,利用检索到的令牌掩码进行稳定的RL训练,并采用简单的基于结果的奖励函数。在七个问答数据集上的实验表明,在相同设置下,Search-R1相比各种RAG基线,性能分别提升了41%(Qwen2.5-7B)和20%(Qwen2.5-3B)。本文进一步提供了关于RL优化方法、LLM选择以及检索增强推理中响应长度动态的实证见解。代码和模型检查点可在https://github.com/PeterGriffinJin/Search-R1获取。

英文摘要:

Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑