Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Department of Computer Science, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Center for Intelligent Information Retrieval, University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校智能信息检索中心);Google Cloud AI Research(谷歌云人工智能研究)
Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院自动化研究所人工智能学院)
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning
Mila - Quebec AI Institute(魁北克AI研究所);Université de Montréal(蒙特利尔大学);Fudan University(复旦大学);McGill University(麦吉尔大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能 chair)
DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning
University of California, Santa Barbara(加州大学圣巴巴拉分校)
研究了大规模知识图谱中的学习推理问题。更具体地说,我们描述了一种新的强化学习框架学习多跳关系路径:我们使用一个基于策略的代理与连续状态的基础上知识图谱嵌入,在KG向量空间的原因,通过采样最有前途的关系,以延长其路径。实验表明,我们提出的方法优于基于路径排序的算法和知识图谱嵌入方法在Freebase和Never-Ending Language Learning数据集上的性能。
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Baichuan Inc.(北川公司);University of Chinese Academy of Sciences(中国科学院大学);Tongji University(同济大学);The University of Edinburgh(爱丁堡大学);Zhejiang University(浙江大学)
ReSearch 提出通过强化学习训练 LLMs 将搜索操作融入推理链,无需监督推理步骤,在 Qwen2.5 系列模型上训练后展现出跨基准的强泛化能力,并自然涌现反思与自我纠错能力。
Coupled Variational Reinforcement Learning for Language Model General Reasoning
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
Virginia Tech(弗吉尼亚理工大学);Emory University(埃默里大学);KAUST(阿联酋卡塔尔大学);Georgia Tech(佐治亚理工学院);Cisco(思科系统);TAMU(德克萨斯大学奥斯汀分校);UT Southwestern Medical Center(德克萨斯西南医学中心);Eigen AI
LoopRPT: Reinforcement Pre-Training for Looped Language Models
Harbin Institute of Technology, Harbin, China(哈尔滨工业大学);Tsinghua University, Beijing, China(清华大学);The Hong Kong University of Science and Technology, Guangzhou, China(香港科学与技术大学)
Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language Models
Harbin Institute of Technology(哈尔滨工业大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Tianjin Normal University(天津师范大学);Pengcheng Laboratory(鹏城实验室)
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学深圳研究院);Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd(中国电信人工智能研究院);College of Integrated Circuits, Zhejiang University, Hangzhou, Zhejiang, China(浙江大学集成电路学院);Zhongguancun Academy, Beijing, China(中关村学院)