通过文本表示引导推理来解锁多模态大语言模型中的空间推理
Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
AI总结：
本文提出TRACE方法，通过生成文本表示来提升多模态大语言模型的空间推理能力，实验表明其在多个基准测试中表现优异。
摘要：
现有多模态大语言模型（MLLMs）在3D空间推理方面存在困难，因为它们无法构建视频输入中3D环境的结构化抽象。为弥合这一差距，我们受认知理论中以自身为中心的空间推理理论启发，研究如何使MLLMs能够建模和推理文本基于的空间表示。具体而言，我们引入了从自身中心视频中提取的文本表示的 allocentric 上下文（TRACE），一种提示方法，该方法促使MLLMs生成3D环境的文本表示作为中间推理轨迹，以更准确地回答空间问题。TRACE 编码元上下文、摄像机轨迹和详细的物体实体，以支持对自身中心视频的结构化空间推理。在VSI-Bench和OST-Bench上的广泛实验表明，TRACE在多种MLLM后端上均表现出显著且一致的改进，涵盖不同参数规模和训练方案。我们进一步进行了消融研究以验证我们的设计选择，并进行了详细分析，探讨了MLLMs在3D空间推理中的瓶颈。
英文摘要：
Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
