负优势是一把双刃剑：为搜索智能体校准GRPO中的优势
Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
AI总结：
针对GRPO算法在多跳搜索中因粗粒度优势分配和正负优势不平衡导致的训练不稳定问题，提出CalibAdv方法，通过细粒度降低过度负优势并重新平衡正负优势，提升模型性能和训练稳定性。
摘要：
搜索智能体通过与搜索引擎的多轮交互实现强大的问答性能，其中组相对策略优化（GRPO）是一种广泛使用的训练算法。然而，GRPO风格的算法在多跳搜索场景中仍面临若干挑战。首先，当最终答案错误时，正确的中间步骤常常受到惩罚。其次，训练高度不稳定，经常导致自然语言能力退化甚至灾难性训练崩溃。我们的分析将这些问题归因于粗粒度的优势分配以及正负优势之间的不平衡。为了解决这些问题，我们提出了CalibAdv，一种专门为搜索智能体设计的优势校准方法，能够更准确、更稳定地对惩罚和奖励进行建模。具体来说，CalibAdv利用中间步骤的正确性在细粒度上降低过度的负优势，然后进一步重新平衡正负优势以提高训练稳定性。重要的是，CalibAdv采用轻量级设计，从标准 rollout 信号中校准优势，使其简单且易于部署。在三个模型和七个基准上的大量实验表明，CalibAdv同时提升了模型性能和训练稳定性。我们的代码可在 https://github.com/wujwyi/CalibAdv 获取。
英文摘要：
Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used training algorithm. However, GRPO-style algorithms still face several challenges in multi-hop search settings. First, correct intermediate steps are often penalized when the final answer is wrong. Second, training is highly unstable, often causing degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for search agents that enables more accurate and more stable modeling of penalties and rewards. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then further rebalances positive and negative advantages to improve training stability. Importantly, CalibAdv adopts a lightweight design that calibrates advantages from standard rollout signals, making it simple and easy to deploy. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at https://github.com/wujwyi/CalibAdv.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
