EgoProx: 在认知层级上评估多模态大语言模型的自我中心3D邻近推理能力
EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy
AI总结：
提出EgoProx基准，通过认知链任务和基于智能体的数据引擎，评估多模态大语言模型在自我中心3D邻近推理中的表现，发现模型虽具备空间知识但难以有效利用。
摘要：
人类不断推理3D邻近性，即身体与周围物体之间的关系，以指导日常生活中的感知和行动。多模态大语言模型（MLLMs）能否进行这种具身3D推理尚不清楚。为此，我们引入了EgoProx，一个用于自我中心3D邻近推理的基准。我们沿着认知链组织任务，涵盖意图、探索、利用和行动链推理。我们还设计了一个基于智能体的数据引擎，能够大规模生成多样且一致的问答对。我们在EgoProx上对主流MLLMs进行了基准测试，并通过数据集特定和任务特定的指令微调进行了额外分析。我们观察到较大的跨领域增益，表明当前的MLLMs包含一些空间知识；然而，它们仍然难以有效利用这些知识进行空间推理VQA。
英文摘要：
Humans constantly reason about 3D proximity, the relations between their body and surrounding objects, to guide perception and action in daily life. Whether multimodal large language models (MLLMs) can perform such embodied 3D reasoning remains unclear. To this end, we introduce EgoProx, a benchmark for egocentric 3D proximity reasoning. We organize our tasks along a cognitive chain, covering intention, exploration, exploitation, and chain-of-actions reasoning. We also design an agent based data engine that produces diverse and consistent QA pairs at scale. We benchmark prevailing MLLMs on EgoProx and conduct additional analyses with dataset specific and task specific instruction tuning. We observe large cross-domain gains, indicating that current MLLMs contain some spatial knowledge; however, they still struggle to effectively leverage it for spatial reasoning VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
