arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10069cs.LGcs.CL
收藏

通过一行代码改进搜索代理

Improving Search Agent with One Line of Code

Nanjing University(南京大学);Tencent YoutuLab(腾讯优图实验室)

Jian Li, Dongsheng Chen, Zhenhua Xu, Yizhang Jin, Jiafu Wu, Chengjie Wang, Xiaotong Yuan, Yabiao Wang

AI总结:

通过一行代码改进搜索代理,提出SAPO算法以稳定训练并提升性能

摘要:

基于工具的代理强化学习(TARL)已成为训练搜索代理以自主与外部工具交互进行多轮信息获取过程的有希望范式。然而,我们发现了一个关键的训练不稳定性问题,导致灾难性模型崩溃:重要性采样分布漂移(ISDD)。在广泛采用的TARL算法Group Relative Policy Optimization(GRPO)中,ISDD表现为重要性采样比率的急剧下降,这会抵消梯度更新并触发不可逆的训练失败。为了解决这个问题,我们提出了Search Agent Policy Optimization(SAPO),通过条件性令牌级KL约束来稳定训练。与硬裁剪不同,后者忽略了分布分歧,SAPO有选择地惩罚当前策略与旧策略之间的KL散度。关键的是,这种惩罚仅应用于概率较低的正向令牌,其中策略已过度偏移,从而在防止分布漂移的同时保持梯度流动。值得注意的是,SAPO仅需对标准GRPO进行一行代码修改,确保了立即的可部署性。在七个QA基准测试中进行了广泛的实验,结果表明,SAPO在Search-R1上实现了+10.6%的绝对改进(+31.5%的相对改进),在不同模型规模(1.5B,14B)和家族(Qwen,LLaMA)上均获得一致的提升。

英文摘要:

Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools for a multi-turn information-seeking process autonomously. However, we identify a critical training instability that leads to catastrophic model collapse: Importance Sampling Distribution Drift(ISDD). In Group Relative Policy Optimization(GRPO), a widely adopted TARL algorithm, ISDD manifests as a precipitous decline in the importance sampling ratios, which nullifies gradient updates and triggers irreversible training failure. To address this, we propose \textbf{S}earch \textbf{A}gent \textbf{P}olicy \textbf{O}ptimization (\textbf{SAPO}), which stabilizes training via a conditional token-level KL constraint. Unlike hard clipping, which ignores distributional divergence, SAPO selectively penalizes the KL divergence between the current and old policies. Crucially, this penalty is applied only to positive tokens with low probabilities where the policy has shifted excessively, thereby preventing distribution drift while preserving gradient flow. Remarkably, SAPO requires only one-line code modification to standard GRPO, ensuring immediate deployability. Extensive experiments across seven QA benchmarks demonstrate that SAPO achieves \textbf{+10.6\% absolute improvement} (+31.5\% relative) over Search-R1, yielding consistent gains across varying model scales (1.5B, 14B) and families (Qwen, LLaMA).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑