arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.01618cs.CVcs.CL
收藏

Actial:激活多模态大语言模型的空间推理能力

Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models

Nanjing University(南京大学);Xiaohongshu Inc.(小红书公司);East China Normal University(华东师范大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);University of Oxford(牛津大学)

Xiaoyu Zhan, Wenxuan Huang, Hao Sun, Xinyu Fu, Changfeng Ma, Shaosheng Cao, Bohan Jia, Shaohui Lin, Zhenfei Yin, Lei Bai, Wanli Ouyang, Yuanqi Li, Jie Guo, Yanwen Guo

AI总结:

本文提出Viewpoint Learning任务与Viewpoint-100K数据集,通过SFT与GRPO两阶段微调及混合冷启动方法,有效激活了多模态大语言模型的空间推理能力,显著提升了其在域内和域外任务上的表现。

摘要:

多模态大语言模型(MLLMs)的最新进展显著提升了2D视觉理解能力,促使人们对其在复杂3D推理任务中的应用产生兴趣。然而,目前尚不清楚这些模型能否有效捕获稳健的真实世界性能所需的详细空间信息,尤其是跨视角一致性,这是准确进行3D推理的关键要求。考虑到这一问题,我们引入了Viewpoint Learning,这是一项旨在评估和提升MLLMs空间推理能力的任务。我们提出了Viewpoint-100K数据集,包含10万个具有不同视角的以对象为中心的图像对及相应的问答对。我们的方法采用两阶段微调策略:首先,通过在Viewpoint-100K上进行监督微调(SFT)向基线MLLM注入基础知识,在多个任务上取得显著提升;其次,通过在更广泛的问题集上使用组相对策略优化(GRPO)算法进行强化学习来增强泛化能力。此外,我们引入了一种混合冷启动初始化方法,旨在同时学习视角表示并保持连贯的推理思考。实验结果表明,我们的方法显著激活了MLLM的空间推理能力,提升了在域内和域外推理任务上的性能。我们的发现强调了在MLLMs中发展基础空间技能的价值,支持了机器人技术、自主系统和3D场景理解领域的未来进展。

英文摘要:

Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved 2D visual understanding, prompting interest in their application to complex 3D reasoning tasks. However, it remains unclear whether these models can effectively capture the detailed spatial information required for robust real-world performance, especially cross-view consistency, a key requirement for accurate 3D reasoning. Considering this issue, we introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs. Our approach employs a two-stage fine-tuning strategy: first, foundational knowledge is injected to the baseline MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in significant improvements across multiple tasks; second, generalization is enhanced through Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm on a broader set of questions. Additionally, we introduce a hybrid cold-start initialization method designed to simultaneously learn viewpoint representations and maintain coherent reasoning thinking. Experimental results show that our approach significantly activates the spatial reasoning ability of MLLM, improving performance on both in-domain and out-of-domain reasoning tasks. Our findings highlight the value of developing foundational spatial skills in MLLMs, supporting future progress in robotics, autonomous systems, and 3D scene understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑