arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.03249cs.CL
收藏

大型语言模型能否成为好的路径规划器?关于时空推理的基准与调查

Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning

George Mason University(乔治梅森大学)

Mohamed Aghzal, Erion Plaku, Ziyu Yao

AI总结:

本研究提出路径规划基准PPNL,评估LLMs时空推理能力,发现少样本GPT-4在空间推理上有潜力但时间推理不足,微调模型泛化能力有限。

摘要:

大型语言模型(LLMs)在广泛的任务中取得了显著成功;然而,在需要长期规划和空间推理的场景中,它们仍然面临局限性。为了促进这一研究方向,在本工作中,我们提出了一个新基准,称为从自然语言进行路径规划(PPNL)。我们的基准通过构建“路径规划”任务来评估LLMs的时空推理能力,这些任务要求LLM在避开障碍物并遵守约束的同时导航至目标位置。利用这一基准,我们系统地调查了包括GPT-4在内的LLMs,通过不同的少样本提示方法,以及不同规模的BART和T5,通过微调。我们的实验结果显示,当GPT-4被提示以交错方式进行推理和行动时,少样本GPT-4在空间推理方面展现出潜力,尽管它仍然无法执行长期时间推理。相比之下,虽然微调后的LLMs在分布内推理任务上取得了令人印象深刻的结果,但它们难以泛化到更大的环境或具有更多障碍物的环境。

英文摘要:

Large language models (LLMs) have achieved remarkable success across a wide spectrum of tasks; however, they still face limitations in scenarios that demand long-term planning and spatial reasoning. To facilitate this line of research, in this work, we propose a new benchmark, termed $\textbf{P}$ath $\textbf{P}$lanning from $\textbf{N}$atural $\textbf{L}$anguage ($\textbf{PPNL}$). Our benchmark evaluates LLMs' spatial-temporal reasoning by formulating ''path planning'' tasks that require an LLM to navigate to target locations while avoiding obstacles and adhering to constraints. Leveraging this benchmark, we systematically investigate LLMs including GPT-4 via different few-shot prompting methodologies as well as BART and T5 of various sizes via fine-tuning. Our experimental results show the promise of few-shot GPT-4 in spatial reasoning, when it is prompted to reason and act interleavedly, although it still fails to perform long-term temporal reasoning. In contrast, while fine-tuned LLMs achieved impressive results on in-distribution reasoning tasks, they struggled to generalize to larger environments or environments with more obstacles.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑