arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.14540cs.CLcs.AI
收藏

大语言模型的空间理解能力评估

Evaluating Spatial Understanding of Large Language Models

Yale University(耶鲁大学);Toyota Technological Institute at Chicago(芝加哥丰田理工学院)

Yutaro Yamada, Yihan Bao, Andrew K. Lampinen, Jungo Kasai, Ilker Yildirim

AI总结:

该研究设计自然语言导航任务,评估GPT-3.5-turbo、GPT-4、Llama2等大语言模型对多种空间结构的表征与推理能力,发现其性能随结构类型差异显著,错误兼具空间与非空间因素,证实LLM隐式掌握部分空间结构但仍有改进空间。

摘要:

大语言模型(LLM)在各类任务中展现出卓越能力。尽管模型在训练过程中仅接触文本,近期多项研究表明LLM的表征隐式捕捉了底层具身概念的多个层面。本文探究LLM对一类尤为重要的具身知识——空间关系——的表征情况。我们设计了自然语言导航任务,评估LLM(尤其是GPT-3.5-turbo、GPT-4和Llama2系列模型)表征与推理空间结构的能力。这些任务揭示出LLM在不同空间结构(包括方格、六边形、三角形网格、环形结构和树形结构)上的性能存在显著差异。通过大量误差分析,我们发现LLM的错误同时反映了空间与非空间因素。这些发现表明,LLM似乎能隐式捕捉空间结构的某些方面,但仍有提升空间。

英文摘要:

Large language models (LLMs) show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge -- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑