Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget.
Narrative Paradoxes in LLM Inference: Shaping Reasoning Trajectories toward Misalignment
School of Electronic, Electrical and Communication Engineering, UCAS(中国科学院大学电子电气与通信工程学院);School of Computer Science and Technology, UCAS(中国科学院大学计算机科学与技术学院);Nanyang Technological University, Singapore(新加坡南洋理工大学);State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全重点实验室);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);Alibaba Group(阿里巴巴集团)
Incorporating planning intelligence into deep learning: A planning support tool for street network design
Martin Centre for Architectural and Urban Studies, University of Cambridge(剑桥大学马丁建筑与城市研究中心);Department of Architecture, University of Cambridge(剑桥大学建筑系)
Prior Prompt Engineering for Reinforcement Fine-Tuning
SCB 10X R&D, SCB 10X, SCBX Group(SCB 10X研发部,SCB 10X,SCBX集团);School of Information Science and Technology, Vidyasirimedhi Institute of Science and Technology(信息科学与技术学院,Vidyasirimedhi科学技术研究院)
Adaptive Science Operations in Deep Space Missions Using Offline Belief State Planning
Department of Aeronautics and Astronautics, Stanford University, Stanford, CA, 94305, USA(斯坦福大学航空航天系);Intelligent Systems Division, NASA Ames Research Center, Moffett Field, CA, 94035, USA(美国航空航天局阿姆斯研究中心智能系统分部)
ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation
University of California, Irvine(加利福尼亚大学欧文分校);University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);Purdue University Northwest(西北普渡大学);Michigan State University(密歇根州立大学);Dartmouth College(达特茅斯学院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Notre Dame(圣母大学)
Learning from Less: Guiding Deep Reinforcement Learning with Differentiable Symbolic Planning
Computer Science Department, TU Darmstadt(图林大学达姆施塔特分校计算机科学系);Hessian Center for Artificial Intelligence (hessian.AI)(黑森人工智能中心);Centre for Cognitive Science, TU Darmstadt(图林大学达姆施塔特分校认知科学中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)