SpaceMind++：迈向空间感知认知地图的视频多模态大语言模型
SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs
AI总结：
SpaceMind++通过构建体素化认知地图，实现空间一致性的视觉推理，提升视频多模态大语言模型在3D环境中的表现。
摘要：
最近的多模态大语言模型（MLLMs）在视觉理解和语言推理方面取得了显著进展，但缺乏用于3D环境中空间一致推理的持久世界中心表示。受哺乳动物双流系统启发，我们提出SpaceMind++，一种视频MLLM架构，通过RGB视频显式构建体素化认知地图。该地图将碎片化的自体中心观察重新组织为共享的3D度量表示，使模型能够在变化的视角下保持物体的持续性和空间拓扑。为了使这种 allocentric 表示能够被预训练的视频MLLM使用而不破坏其原有的视觉令牌接口，我们引入了坐标引导深度迭代融合机制，该机制通过坐标嵌入和3D旋转位置编码将地图级别的空间知识反馈到原始2D视觉特征中。这种融合明确由坐标嵌入和3D旋转位置编码引导，使语义交互在度量3D空间中扎根，类似于嗅觉皮层将感觉特征绑定到度量空间。大量实验表明，SpaceMind++在VSI-Bench上实现了新的SOTA性能。此外，它在SPBench、SITE-Bench和SPAR-Bench上展示了优越的分布外泛化能力，证明了其在未见3D环境中的鲁棒性。
英文摘要：
Recent multimodal large language models (MLLMs) have made remarkable progress in visual understanding and language-based reasoning, yet they lack a persistent world-centered representation for spatially consistent reasoning in 3D environments. Inspired by the mammalian dual-stream system, where semantic and spatial cues are processed separately and integrated into an allocentric cognitive map, we propose SpaceMind++, a video MLLM architecture that explicitly builds a voxelized cognitive map from RGB videos. This map reorganizes fragmented egocentric observations into a shared 3D metric representation, enabling the model to preserve object permanence and spatial topology across changing viewpoints. To make this allocentric representation usable by a pretrained video MLLM without disrupting its native visual-token interface, we introduce Coordinate-Guided Deep Iterative Fusion, a new mechanism that relays map-level spatial knowledge back into the original 2D visual features. This fusion is explicitly guided by coordinate embeddings and 3D Rotary Positional Encoding, which ground semantic interactions in metric 3D space, resembling the entorhinal binding of sensory features to metric space. Extensive experiments show that SpaceMind++ achieves new state-of-the-art performance on VSI-Bench. Furthermore, it demonstrates superior out-of-distribution generalization on SPBench, SITE-Bench, and SPAR-Bench, underscoring its robustness in unseen 3D environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
