arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14992cs.CLcs.AIcs.LG
收藏

基于语言模型的推理即基于世界模型的规划

Reasoning with Language Model is Planning with World Model

UC San Diego(加州大学圣地亚哥分校);University of Florida(佛罗里达大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu

AI总结:

本文提出了一种名为RAP的新型LLM推理框架,通过将LLM同时用作世界模型和推理代理,并结合基于蒙特卡洛树搜索的规划算法,显著提升了LLM在规划生成、数学推理和逻辑推理等复杂任务上的表现,甚至以LLAMA-33B超越了GPT-4上的CoT。

摘要:

大型语言模型(LLMs)展现出了卓越的推理能力,尤其是在被提示生成中间推理步骤(例如,Chain-of-Thought, CoT)时。然而,LLMs在处理对人类来说很容易的问题时仍可能遇到困难,例如为在给定环境中执行任务生成行动计划,或进行复杂的数学、逻辑和常识推理。这种不足源于一个关键事实:LLMs缺乏一个内部世界模型来预测世界状态(例如,环境状态、中间变量值)并模拟行动的长期结果。这阻碍了LLMs执行类似于人脑的深思熟虑的规划,这种规划涉及探索替代推理路径、预测未来状态和奖励,以及迭代地完善现有推理步骤。为了克服这些限制,我们提出了一种新的LLM推理框架,即通过规划进行推理(RAP)。RAP将LLM重新用作世界模型和推理代理,并引入了一种基于蒙特卡洛树搜索的原则性规划算法,用于在广阔的推理空间中进行战略性探索。在推理过程中,LLM(作为代理)在LLM(作为世界模型)和特定任务奖励的指导下逐步构建推理树,并在探索与利用之间取得适当平衡,从而高效地获得高奖励的推理路径。我们将RAP应用于多种具有挑战性的推理问题,包括规划生成、数学推理和逻辑推理。在这些任务上的实证结果证明了RAP优于各种强大的基线方法,包括CoT和带有自洽性的从少到多提示。在规划生成设置中,LLAMA-33B上的RAP超越了GPT-4上的CoT,实现了33%的相对提升。

英文摘要:

Large language models (LLMs) have shown remarkable reasoning capabilities, especially when prompted to generate intermediate reasoning steps (e.g., Chain-of-Thought, CoT). However, LLMs can still struggle with problems that are easy for humans, such as generating action plans for executing tasks in a given environment, or performing complex math, logical, and commonsense reasoning. The deficiency stems from the key fact that LLMs lack an internal $\textit{world model}$ to predict the world $\textit{state}$ (e.g., environment status, intermediate variable values) and simulate long-term outcomes of actions. This prevents LLMs from performing deliberate planning akin to human brains, which involves exploring alternative reasoning paths, anticipating future states and rewards, and iteratively refining existing reasoning steps. To overcome the limitations, we propose a new LLM reasoning framework, $\underline{R}$easoning vi$\underline{a}$ $\underline{P}$lanning $\textbf{(RAP)}$. RAP repurposes the LLM as both a world model and a reasoning agent, and incorporates a principled planning algorithm (based on Monto Carlo Tree Search) for strategic exploration in the vast reasoning space. During reasoning, the LLM (as agent) incrementally builds a reasoning tree under the guidance of the LLM (as world model) and task-specific rewards, and obtains a high-reward reasoning path efficiently with a proper balance between exploration $\textit{vs.}$ exploitation. We apply RAP to a variety of challenging reasoning problems including plan generation, math reasoning, and logical inference. Empirical results on these tasks demonstrate the superiority of RAP over various strong baselines, including CoT and least-to-most prompting with self-consistency. RAP on LLAMA-33B surpasses CoT on GPT-4 with 33% relative improvement in a plan generation setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑