EgoMind: 通过多模态大语言模型中的语言推理激活空间认知
EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
AI总结:
提出EgoMind框架,通过角色扮演描述和渐进空间分析的无几何空间推理方法,仅用少量数据即可在多基准测试中提升MLLMs的空间推理能力。
摘要:
多模态大语言模型(MLLMs)越来越多地应用于空间认知任务,期望它们能够理解并与复杂环境交互。现有工作大多通过引入3D先验或几何监督来改进空间推理,这虽然提升了性能,但带来了大量的数据准备和对齐成本。相比之下,纯2D方法由于捕获跨帧空间关系的能力有限,往往在多帧空间推理中表现不佳。为了解决这些限制,我们提出了EgoMind,一个思维链框架,通过角色扮演描述(联合构建跨帧的一致语言场景图)和渐进空间分析(逐步推理任务特定问题)实现无几何空间推理。仅使用5K自动生成的SFT样本和20K RL样本,EgoMind在VSI-Bench、SPAR-Bench、SITE-Bench和SPBench上取得了有竞争力的结果,证明了其在增强MLLMs空间推理能力方面的有效性,并突出了语言推理在空间认知中的潜力。代码和数据已发布在https://github.com/Hyggge/EgoMind。
英文摘要:
Multimodal large language models (MLLMs) are increasingly being applied to spatial cognition tasks, where they are expected to understand and interact with complex environments. Most existing works improve spatial reasoning by introducing 3D priors or geometric supervision, which enhances performance but incurs substantial data preparation and alignment costs. In contrast, purely 2D approaches often struggle with multi-frame spatial reasoning due to their limited ability to capture cross-frame spatial relationships. To address these limitations, we propose EgoMind, a Chain-of-Thought framework that enables geometry-free spatial reasoning through Role-Play Caption, which jointly constructs a coherent linguistic scene graph across frames, and Progressive Spatial Analysis, which progressively reasons toward task-specific questions. With only 5K auto-generated SFT samples and 20K RL samples, EgoMind achieves competitive results on VSI-Bench, SPAR-Bench, SITE-Bench, and SPBench, demonstrating its effectiveness in strengthening the spatial reasoning capabilities of MLLMs and highlighting the potential of linguistic reasoning for spatial cognition. Code and data are released at https://github.com/Hyggge/EgoMind.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。