ARBOR: 通过可复用评分标准缓冲区的在线过程奖励用于搜索智能体
ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
AI总结:
针对基于LLM的搜索智能体训练中结果奖励缺乏过程监督的问题,提出ARBOR框架,通过维护跨查询共享的评分标准记忆库,利用对比轨迹生成局部草案并整合为通用评分标准,以稀疏成对判断提供过程级梯度,在四个多跳QA基准上优于GRPO和DAPO基线。
摘要:
基于LLM的搜索智能体主要使用结果奖励进行训练,搜索过程本身无监督。这种信号在结果同质组(所有采样轨迹共享相同正确性)中退化,产生零组内优势和无梯度。现有的过程监督要么训练昂贵的验证器,要么生成每个查询的评分标准,这些评分标准在查询间不一致且使用一次后丢弃。我们提出ARBOR(自适应评分标准缓冲区用于在线奖励),一种可复用的过程奖励框架,维护跨查询共享的评分标准记忆。由对比轨迹诱导的查询局部草案被接纳、整合为跨查询通用评分标准,并随策略演化而淘汰。一小部分活跃的通用评分标准通过稀疏成对判断对轨迹评分,所得分数加到基础奖励上,即使在结果奖励一致时也能提供过程级梯度。ARBOR在四个多跳QA基准上持续优于GRPO和DAPO基线,将LLM评判准确率平均提高最多4.2个百分点,并将最多42%的原本零梯度训练组转化为信息丰富的组。
英文摘要:
LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised. This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient. Existing process supervision either trains a costly verifier or generates per-query rubrics that are inconsistent across queries and discarded after one use. We propose ARBOR (Adaptive Rubric Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries. Query-local drafts induced from contrastive trajectories are admitted, consolidated into cross-query common rubrics, and retired as the policy evolves. A small active subset of common rubrics scores trajectories via sparse pairwise judging, and the resulting scores are added to the base reward, providing process-level gradient even when outcome reward is uniform. ARBOR consistently outperforms GRPO and DAPO baselines on four multi-hop QA benchmarks, raising average LLM-judge accuracy by up to 4.2 points and converting up to 42% of otherwise-zero-gradient training groups into informative ones.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。