arXiv 2605.116112026-08-12cs.AI

CuSearch：通过搜索深度进行课程展开采样以实现代理RAG

CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAG

Nanjing University（南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University（北京大学）；University of Hong Kong（香港大学）

Jianghan Shen, Siqi Luo, Xinyu Cheng, Jing Xiong, Yue Li, Jiyao Liu, Jiashi Lin, Yirong Chen, Junjun He

备注：: The paper has been accepted by COLM 2026. Code: https://github.com/MrToser/CuSearch

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CuSearch，通过搜索深度贪心分配策略改进代理RAG训练，提升检索监督密度，实验显示在ZeroSearch上性能提升达11.8个精确匹配点。

摘要：

可验证奖励的强化学习（RLVR）已发展为从仅结果监督训练代理检索增强生成（RAG）系统的一种有前途的范式。现有方法通过均匀采样的展开式优化策略，隐含地将所有轨迹视为同等信息量。然而，轨迹在搜索深度上差异显著，因此信息量不均：深度搜索轨迹包含更多检索决策点，为检索子策略提供更密集的直接监督。此外，这种异质性随训练增加，随着批次内深度分布向更高值偏移，但均匀展开采样仍无法察觉这一变化。为此，我们提出CuSearch，一种基于搜索深度贪心分配（SDGA）的课程展开采样框架，是一种批级操作，将固定更新预算重新分配给更深层次的轨迹。SDGA-Auto始终瞄准当前批次中可用的最深轨迹，产生隐含的与深度分布偏移对齐的课程。SDGA-Phase则显式地将课程阈值提升，当更深层的轨迹变得足够充足时。实验表明，CuSearch在不同模型类型和检索框架上均提升了性能，其在ZeroSearch上相对于标准GRPO的精确匹配点提升达11.8个。这些结果确立了每轨迹搜索深度作为RLVR代理RAG训练中检索监督密度的可靠、无标注代理。

英文摘要：

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for training agentic retrieval-augmented generation (RAG) systems from outcome-only supervision. Most existing methods optimize policies from uniformly sampled rollouts, implicitly treating all trajectories as equally informative. However, trajectories differ substantially in search depth and are therefore not equally informative: deeper-search trajectories contain more retrieval decision points and provide denser direct supervision for the retrieval sub-policy. Moreover, this heterogeneity grows over training as the within-batch depth distribution shifts toward higher values, yet uniform rollout sampling remains blind to this shift. To address this, we propose CuSearch, a curriculum rollout sampling framework built on Search-Depth Greedy Allocation (SDGA), a batch-level operator that reallocates a fixed update budget toward deeper-search trajectories. SDGA-Auto always targets the deepest available trajectories in the current batch, yielding an implicit training-aligned curriculum as the depth distribution shifts upward. SDGA-Phase explicitly advances the curriculum threshold as deeper trajectories become sufficiently abundant. Experiments across model types and retrieval frameworks show that CuSearch consistently improves performance, achieving up to 11.8 exact-match points over standard GRPO on ZeroSearch. These results establish per-trajectory search depth as a reliable, annotation-free proxy for retrieval supervision density in RLVR-based agentic RAG training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。