通过贡献加权群体相对策略优化增强基于LLM的搜索代理
Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
AI总结：
本文提出CW-GRPO框架，通过整合过程监督与群体相对策略优化，提升搜索代理的信用分配效率，实验表明其在多个知识密集型基准上表现优异。
摘要：
搜索代理通过扩展大型语言模型（LLM）的知识范围，使代理能够访问到训练时不可用的最新和长尾信息。尽管强化学习已被广泛用于训练此类代理，但现有方法面临关键限制：过程监督常因价值估计不稳定而受限，而结果监督则因稀疏的轨迹级奖励导致信用分配困难。为此，我们提出贡献加权群体相对策略优化（CW-GRPO）框架，该框架将过程监督整合到群体相对策略优化中。CW-GRPO不直接优化过程奖励，而是利用LLM法官在每次搜索轮次中评估检索效用和推理正确性，生成每轮贡献评分。这些评分用于沿轨迹重新缩放基于结果的优势，从而实现细粒度的信用分配而不牺牲优化稳定性。在多个知识密集型基准上的实验表明，CW-GRPO在Qwen3-8B上比标准GRPO高出5.0%，在Qwen3-1.7B上高出6.3%，从而产生更有效的搜索行为。进一步分析表明，成功轨迹在特定轮次中表现出集中的贡献，为搜索代理任务提供了实证见解。
英文摘要：
Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
