arXiv 2505.207532025-05-27cs.CVcs.AI

理解、思考与回答：利用大型多模态模型推进视觉推理

Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）

Yufei Zhan, Hongyin Zhao, Yousong Zhu, Shurong Zheng, Fan Yang, Ming Tang, Jinqiao Wang

备注：: Tech report

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种基于理解-思考-回答过程的统一视觉推理机制，使大型多模态模型能单次前向传播解决复杂组合问题，并训练了Griffon-R模型，在VSR、CLEVR等基准上取得先进性能。

摘要：

大型多模态模型（LMMs）近期在视觉-语言任务和以视觉为中心的任务上展示了显著的视觉理解性能。然而，它们往往在整合高级的、任务特定的组合推理能力方面存在不足，这阻碍了它们向真正胜任的通用视觉模型发展。为解决这一问题，我们提出了一种统一的视觉推理机制，使LMMs能够通过利用其内在能力（如定位和视觉理解能力）来解决复杂的组合问题。与之前的捷径学习机制不同，我们的方法引入了一个类似人类的“理解-思考-回答”过程，使模型能够在单次前向传播中完成所有步骤，无需多次推理或外部工具。这种设计弥合了基础视觉能力与通用问答之间的差距，鼓励LMMs为复杂的视觉推理生成忠实且可追溯的响应。同时，我们整理了334K个视觉指令样本，涵盖一般场景和文本丰富场景，并涉及多种基础视觉能力。我们训练的模型Griffon-R具备端到端的自动理解、自我思考和推理回答能力。综合实验表明，Griffon-R不仅在包括VSR和CLEVR在内的复杂视觉推理基准上取得了先进性能，还增强了在MMBench和ScienceQA等多个基准上的多模态能力。数据、模型和代码将很快在https://github.com/jefferyZhan/Griffon/tree/master/Griffon-R发布。

英文摘要：

Large Multimodal Models (LMMs) have recently demonstrated remarkable visual understanding performance on both vision-language and vision-centric tasks. However, they often fall short in integrating advanced, task-specific capabilities for compositional reasoning, which hinders their progress toward truly competent general vision models. To address this, we present a unified visual reasoning mechanism that enables LMMs to solve complicated compositional problems by leveraging their intrinsic capabilities (e.g. grounding and visual understanding capabilities). Different from the previous shortcut learning mechanism, our approach introduces a human-like understanding-thinking-answering process, allowing the model to complete all steps in a single pass forwarding without the need for multiple inferences or external tools. This design bridges the gap between foundational visual capabilities and general question answering, encouraging LMMs to generate faithful and traceable responses for complex visual reasoning. Meanwhile, we curate 334K visual instruction samples covering both general scenes and text-rich scenes and involving multiple foundational visual capabilities. Our trained model, Griffon-R, has the ability of end-to-end automatic understanding, self-thinking, and reasoning answers. Comprehensive experiments show that Griffon-R not only achieves advancing performance on complex visual reasoning benchmarks including VSR and CLEVR, but also enhances multimodal capabilities across various benchmarks like MMBench and ScienceQA. Data, models, and codes will be release at https://github.com/jefferyZhan/Griffon/tree/master/Griffon-R soon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。