arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03647cs.AIcs.CL
收藏

Search-R2: 通过演员-细化协作提升搜索集成推理

Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration

The Chinese University of Hong Kong(香港中文大学);LLM Department, Tencent(腾讯语言模型部门);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);McGill University(麦吉尔大学);City University of Hong Kong(香港城市大学);The University of Edinburgh(爱丁堡大学)

Bowei He, Minda Hu, Zenan Xu, Hongru Wang, Licheng Zong, Yankai Chen, Chen Ma, Xue Liu, Pluto Zhou, Irwin King

AI总结:

Search-R2通过演员-细化协作框架提升搜索集成推理,通过针对性干预和混合奖励设计,在多种QA数据集上超越强基线,实现更优的推理准确性。

摘要:

搜索集成推理使语言代理能够超越静态参数化知识,通过主动查询外部来源。然而,通过强化学习训练这些代理受到多尺度信用分配问题的阻碍:现有方法通常依赖于稀疏的轨迹级奖励,无法区分高质量推理和偶然猜测,导致冗余或误导的搜索行为。为此,我们提出了Search-R2,一种新颖的演员-细化协作框架,通过针对性干预提升推理能力,两个组件在训练过程中共同优化。我们的方法将生成过程分解为一个演员,生成初始推理轨迹,以及一个元细化器,通过'剪切并重新生成'机制选择性地诊断和修复错误步骤。为了提供细粒度监督,我们引入了一种混合奖励设计,将结果正确性与量化检索证据信息密度的密集过程奖励相结合。理论上,我们将演员-细化器交互形式化为一个平滑混合策略,证明选择性修正在强基线之上产生严格的表现提升。在各种通用和多跳QA数据集上的广泛实验表明,Search-R2在模型规模上一致优于强RAG和RL基线,以最小的开销实现更优的推理准确性。

英文摘要:

Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a 'cut-and-regenerate' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑