证据链的构建：基于引用感知的鲁棒深度搜索代理强化学习
Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards
AI总结：
本文提出引用感知评分框架和组相对策略优化方法，用于提升深度搜索代理的鲁棒性和推理质量。
摘要：
强化学习（RL）已成为增强基于大语言模型（LLM）的深度搜索代理的关键技术。然而，现有方法主要依赖二元结果奖励，这无法捕捉代理推理过程的全面性和事实性，往往导致不期望的行为，如捷径利用和幻觉。为了解决这些限制，我们提出了引用感知的评分框架（CaRR），这是一种针对深度搜索代理的细粒度奖励框架，强调推理的全面性、事实基础和证据连接性。CaRR将复杂问题分解为可验证的单跳评分标准，并要求代理通过显式识别隐藏实体、用正确引用支持这些实体，并构建完整的证据链来连接到预测答案。我们进一步引入了引用感知的组相对策略优化（C-GRPO），将CaRR与结果奖励结合，以训练鲁棒的深度搜索代理。实验表明，C-GRPO在多个深度搜索基准上均优于标准结果基于的RL基线。我们的分析也验证了C-GRPO有效遏制了捷径利用，促进了全面、证据基础的推理，并在开放性深度研究任务中表现出强泛化能力。我们的代码和数据可在https://github.com/THUDM/CaRR上获得。
英文摘要：
Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents. However, existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process, and often lead to undesirable behaviors such as shortcut exploitation and hallucinations. To address these limitations, we propose \textbf{Citation-aware Rubric Rewards (CaRR)}, a fine-grained reward framework for deep search agents that emphasizes reasoning comprehensiveness, factual grounding, and evidence connectivity. CaRR decomposes complex questions into verifiable single-hop rubrics and requires agents to satisfy these rubrics by explicitly identifying hidden entities, supporting them with correct citations, and constructing complete evidence chains that link to the predicted answer. We further introduce \textbf{Citation-aware Group Relative Policy Optimization (C-GRPO)}, which combines CaRR and outcome rewards for training robust deep search agents. Experiments show that C-GRPO consistently outperforms standard outcome-based RL baselines across multiple deep search benchmarks. Our analysis also validates that C-GRPO effectively discourages shortcut exploitation, promotes comprehensive, evidence-grounded reasoning, and exhibits strong generalization to open-ended deep research tasks. Our code and data are available at https://github.com/THUDM/CaRR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
