arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09255cs.CL
收藏

DSPO:用于代理搜索和推理的稳定高效策略优化

DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning

Sapiens AI;Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学)

Chenyang Gu, Yewen Pu, Bruce Yang, Xiaofan Li, Huan Gao

AI总结:

DSPO通过序列级优化和动态样本过滤提升代理训练稳定性,使7B模型在多个问答基准上超越前作34.1%,在复杂多跳问答中甚至超越14B模型9个百分点。

摘要:

增强大语言模型主动搜索外部知识的能力对复杂现实任务至关重要。当前方法或依赖提示提取模型固有代理能力,或在应用RL到复杂交互任务时遭遇性能瓶颈。为此,我们提出DSPO,一种改进的RL算法,通过序列级优化和动态样本过滤实现稳健的代理训练。我们通过纯RL训练模型,交错多轮搜索与推理,无需监督演示数据。在多个问答基准上,我们的7B模型在多个问答基准上超越了可比的先前工作34.1%,并在复杂多跳问答如HotpotQA中甚至超越了先前工作的14B模型近9个百分点,保持了出色的训练稳定性。

英文摘要:

Enhancing LLMs with the ability to actively search external knowledge is crucial for complex and real-world tasks. Current approaches either rely on prompting to elicit the model's innate agent capabilities, or suffer from performance ceilings and collapse when applying RL to complex interactive tasks, leaving their true agentic potential untapped. To address this, we introduce \textbf{D}ynamic-filter \textbf{S}equence-level \textbf{P}olicy \textbf{O}ptimization (DSPO), an improved RL algorithm designed for robust agent training through sequence-level optimization and dynamic sample filtering. We train our model purely through RL to interleave multi-turn search and reasoning, obviating the need for supervised demonstration data. Across multiple QA benchmarks, our 7B model improves over a comparable previous work by \textbf{34.1\%}, and even outperforms the 14B model from previous work in complex multihop QA such as HotpotQA by nearly \textbf{9\% relative}, maintaining exceptional training stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑