arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.12979cs.AIcs.CL
收藏

DeepPlanner:通过优势塑造提升深度研究智能体的规划能力

DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping

The Hong Kong University of Science and Technology(香港科技大学);Amazon(亚马逊);University of California, San Diego(加州大学圣地亚哥分校)

Wei Fan, Wenlin Yao, Zheng Li, Feng Yao, Xin Liu, Liang Qiu, Qingyu Yin, Yangqiu Song, Bing Yin

AI总结:

针对现有深度研究智能体规划阶段优化不足的问题,提出端到端 RL 框架 DeepPlanner,通过塑造 token 级优势与选择性提升样本级优势,在7个基准上以更低训练预算实现最优规划效果。

摘要:

结合多步推理与动作生成能力的大语言模型(LLMs),已展现出利用外部工具解决需要长时序规划的复杂任务的潜力。然而,现有方法要么依赖推理阶段的隐式规划,要么引入显式规划器却未系统解决规划阶段的优化问题。我们观察到,在标准强化学习(RL)设置下,规划 token 的熵显著高于其他动作 token,暴露出仍未充分优化的不确定决策点。为解决该问题,我们提出 DeepPlanner,一种可有效增强深度研究智能体规划能力的端到端 RL 框架。该方法通过基于熵的项塑造 token 级优势,为高熵 token 分配更大更新量,并选择性提升规划密集型 rollout 的样本级优势。在7个深度研究基准上开展的大量实验表明,DeepPlanner 可提升规划质量,并在显著更低的训练预算下达到了当前最优结果。

英文摘要:

Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑