少些对话，多些行动：探究大语言模型在3D具身环境中的物理常识
A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment
AI总结：
本文提出在 Animal-AI 3D 具身环境中评估大语言模型的物理常识推理能力，证明无需微调的多模态模型可完成距离估计、追踪视线外物体和工具使用等任务，但其表现目前仍不及人类儿童。
摘要：
作为通用工具，大语言模型（LLMs）常常需要对日常物理环境进行推理。在问答场景中，理解物理对象之间的相互作用可能是给出恰当回答的必要条件。此外，LLMs 越来越多地被用作智能体系统中的推理引擎，用于设计和控制其动作序列。绝大多数研究通过静态基准来处理这一问题，这些基准由关于物理世界的文本或图像问题组成。然而，这些基准无法捕捉现实物理过程的复杂性和细微差别。在此，我们倡导第二种相对未被探索的方法：通过赋予 LLMs 对 3D 环境中智能体的控制权，使其“具身化”。我们提出了首个针对 LLMs 物理常识推理的具身且具有认知意义的评估。我们的框架允许将 LLMs 与其他具身智能体（例如基于深度强化学习的智能体）以及人类和非人类动物进行直接比较。我们采用 Animal-AI（AAI）环境——一个模拟的 3D 虚拟实验室——来研究 LLMs 的物理常识推理能力。为此，我们使用 AAI Testbed，这是一套复刻非人类动物实验室研究的实验组合，用于研究物理推理能力，包括距离估计、追踪视线外物体以及工具使用。我们证明，无需微调的最先进多模态模型即可完成此类任务，从而能够与 2019 年 Animal-AI Olympics 竞赛的参赛者以及人类儿童进行有意义的比较。我们的结果表明，LLMs 目前在这些任务上的表现不如人类儿童。我们认为，这种方法允许使用直接源自认知科学的生态有效实验来研究物理推理，从而提高 LLMs 的可预测性和可靠性。
英文摘要：
As general-purpose tools, Large Language Models (LLMs) must often reason about everyday physical environments. In a question-and-answer capacity, understanding the interactions of physical objects may be necessary to give appropriate responses. Moreover, LLMs are increasingly used as reasoning engines in agentic systems, designing and controlling their action sequences. The vast majority of research has tackled this issue using static benchmarks, comprised of text or image-based questions about the physical world. However, these benchmarks do not capture the complexity and nuance of real-life physical processes. Here we advocate for a second, relatively unexplored, approach: 'embodying' the LLMs by granting them control of an agent within a 3D environment. We present the first embodied and cognitively meaningful evaluation of physical common-sense reasoning in LLMs. Our framework allows direct comparison of LLMs with other embodied agents, such as those based on Deep Reinforcement Learning, and human and non-human animals. We employ the Animal-AI (AAI) environment, a simulated 3D virtual laboratory, to study physical common-sense reasoning in LLMs. For this, we use the AAI Testbed, a suite of experiments that replicate laboratory studies with non-human animals, to study physical reasoning capabilities including distance estimation, tracking out-of-sight objects, and tool use. We demonstrate that state-of-the-art multi-modal models with no finetuning can complete this style of task, allowing meaningful comparison to the entrants of the 2019 Animal-AI Olympics competition and to human children. Our results show that LLMs are currently outperformed by human children on these tasks. We argue that this approach allows the study of physical reasoning using ecologically valid experiments drawn directly from cognitive science, improving the predictability and reliability of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
