SpatialGeo:通过几何-语义融合提升多模态大语言模型的空间推理能力
SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion
AI总结：
SpatialGeo通过几何-语义融合提升多模态大语言模型的空间推理能力，实验表明在空间推理任务中准确率提升8.0%且内存消耗减少50%。
摘要：
多模态大语言模型（MLLMs）由于大语言模型（LLMs）强大的推理能力，在图像和语言任务中取得了显著进展。然而，大多数MLLMs在解释和推断三维空间中的空间安排时存在空间推理能力有限的问题。在本工作中，我们提出了一种基于层次融合几何和语义特征的新视觉编码器，生成空间感知的视觉嵌入，并提升MLLMs的空间定位能力。具体来说，我们首先揭示了空间模糊性缺陷源于现有大多数MLLMs（如CLIP）中使用的视觉编码器的损失性嵌入，受限于实例级语义特征。这促使我们通过层次化适配器补充CLIP的几何特征，通过视觉-only自监督学习，增强所提出SpatialGeo中的空间感知能力。网络通过预训练LLaVA模型高效训练，并通过随机特征丢弃进行优化，以避免仅依赖CLIP编码器的平凡解。实验结果表明，SpatialGeo在空间推理任务中提高了准确性，至少在SpatialRGPT-Bench上比最先进的模型提升了8.0%，并且在推理过程中内存消耗减少了约50%。源代码可通过https://ricky-plus.github.io/SpatialGeoPages/获取。
英文摘要：
Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with approximately 50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
