arXiv 2508.061652026-09-22cs.CLcs.AI

UR$^2$：通过强化学习统一检索增强生成与推理

UR$^2$: Unify RAG and Reasoning through Reinforcement Learning

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学，北京，中国）；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China（人工智能产业研究机构（AIR），清华大学，北京，中国）；School of Management Science & Information Engineering, Hebei University of Economics and Business, Hebei, China（管理科学与信息工程学院，河北经贸大学，河北，中国）

Weitao Li, Boran Xiang, Xiaolong Wang, Jingyi Ren, Ante Wang, Zhinan Gou, Weizhi Ma, Yang Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出UR$^2$框架，通过强化学习动态协调检索与推理，结合难度感知课程和混合知识访问策略，在开放域问答、MMLU-Pro、医学和数学推理任务上优于现有基线，性能接近GPT-4o-mini和GPT-4.1-mini。

摘要：

大型语言模型（LLM）通过两种互补范式展现了强大能力：用于知识基础的检索增强生成（RAG）和用于复杂推理的可验证奖励强化学习（RLVR）。然而，现有统一这些范式的尝试范围狭窄，通常局限于具有固定检索设置的开放域问答，限制了向更广泛领域的泛化。为解决这一局限，我们提出UR$^2$（统一RAG与推理），一个通用的强化学习框架，动态协调检索与推理。UR$^2$引入了两个关键设计：一个难度感知课程，仅对困难实例选择性调用检索；以及一个混合知识访问策略，结合领域特定的离线语料库和即时生成的LLM摘要。这些组件共同缓解了检索与推理之间的不平衡，并提高了对噪声信息的鲁棒性。在开放域问答、MMLU-Pro、医学和数学推理任务上的实验表明，基于Qwen-2.5-3/7B和LLaMA-3.1-8B构建的UR$^2$持续优于现有RAG和RL基线，并在多个基准上达到与GPT-4o-mini和GPT-4.1-mini相当的性能。我们的代码可在https://github.com/Tsinghua-dhy/UR2获取。

英文摘要：

Large Language Models (LLMs) have shown strong capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG) for knowledge grounding and Reinforcement Learning from Verifiable Rewards (RLVR) for complex reasoning. However, existing attempts to unify these paradigms remain narrow in scope, typically limited to open-domain QA with fixed retrieval settings, which constrains generalization to broader domains. To address this limitation, we propose UR$^2$ (Unified RAG and Reasoning)), a general reinforcement learning framework that dynamically coordinates retrieval and reasoning. UR$^2$ introduces two key designs: a difficulty-aware curriculum that selectively invokes retrieval only for challenging instances, and a hybrid knowledge access strategy that combines domain-specific offline corpora with on-the-fly LLM-generated summaries. Together, these components mitigate the imbalance between retrieval and reasoning and improve robustness to noisy information. Experiments on open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks show that UR$^2$, built on Qwen-2.5-3/7B and LLaMA-3.1-8B, consistently outperforms existing RAG and RL baselines, and achieves performance comparable to GPT-4o-mini and GPT-4.1-mini on several benchmarks. Code, models, and data are available at https://github.com/Tsinghua-dhy/UR2.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。