Large language models (LLMs) have shown remarkable reasoning capabilities, especially when prompted to generate intermediate reasoning steps (e.g., Chain-of-Thought, CoT). However, LLMs can still struggle with problems that are easy for humans, such as generating action plans for executing tasks in a given environment, or performing complex math, logical, and commonsense reasoning. The deficiency stems from the key fact that LLMs lack an internal $\textit{world model}$ to predict the world $\textit{state}$ (e.g., environment status, intermediate variable values) and simulate long-term outcomes of actions. This prevents LLMs from performing deliberate planning akin to human brains, which involves exploring alternative reasoning paths, anticipating future states and rewards, and iteratively refining existing reasoning steps. To overcome the limitations, we propose a new LLM reasoning framework, $\underline{R}$easoning vi$\underline{a}$ $\underline{P}$lanning $\textbf{(RAP)}$. RAP repurposes the LLM as both a world model and a reasoning agent, and incorporates a principled planning algorithm (based on Monto Carlo Tree Search) for strategic exploration in the vast reasoning space. During reasoning, the LLM (as agent) incrementally builds a reasoning tree under the guidance of the LLM (as world model) and task-specific rewards, and obtains a high-reward reasoning path efficiently with a proper balance between exploration $\textit{vs.}$ exploitation. We apply RAP to a variety of challenging reasoning problems including plan generation, math reasoning, and logical inference. Empirical results on these tasks demonstrate the superiority of RAP over various strong baselines, including CoT and least-to-most prompting with self-consistency. RAP on LLAMA-33B surpasses CoT on GPT-4 with 33% relative improvement in a plan generation setting.
Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents
University of Notre Dame(诺丁汉大学);Stanford University(斯坦福大学);Purdue University(普渡大学);University of Edinburgh(爱丁堡大学);University of Oxford(牛津大学);Yale University(耶鲁大学)
Non-myopic Generation of Language Models for Reasoning and Planning
The University of Hong Kong(香港大学);Peking University(北京大学);Zhejiang University(浙江大学);Westlake University(西湖大学);The Hong Kong University of Science and Technology(香港科技大学)
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
S-Lab, Nanyang Technological University, Singapore(南洋理工大学S实验室);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院)
IMAGINE: Integrating Multi-Agent System into One Model for Complex Reasoning and Planning
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Kuaishou Technology(快手科技)
Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
Case Western Reserve University, Cleveland, OH, USA(凯斯西储大学);Kean University, Union, NJ, USA(凯恩大学);The Ohio State University, Columbus, OH, USA(俄亥俄州立大学);Fudan University, Shanghai, China(复旦大学);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室);The University of Hong Kong, Hong Kong, China(香港大学);North Carolina State University, Raleigh, NC, USA(北卡罗来纳州立大学)
PRIME: Planning and Retrieval-Integrated Memory for Enhanced Reasoning
Center for Healthcare Organization and Implementation Research, VA Bedford Health Care(VA Bedford Health Care 医疗组织与实施研究中心);Manning College of Information and Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校信息与计算机科学学院);Miner School of Computer and Information Sciences, University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校 Miner 信息与计算机科学学院)
HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds
Artyom Sorokin(AXXX);Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Higher School of Economics(高等经济学院);Kurchatov Institute(库尔斯克研究所);Independent Researcher(独立研究者)
Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学);The Australian National University(澳大利亚国立大学);Fudan University(复旦大学);University of Sydney(悉尼大学)
Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Models
University of Maryland, College Park MD, USA(马里兰大学);ERA Lab, University of South Florida, Tampa FL, USA(佛罗里达大学埃拉实验室);DEVCOM Army Research Laboratory, Aberdeen Proving Ground MD, USA(国防部陆军研究实验室);EEHPC Lab, Johns Hopkins University, Baltimore MD, USA(约翰霍普金斯大学EEHPC实验室)