arXiv 2506.010782025-06-01cs.CVcs.AI

GThinker：通过线索引导的再思考实现通用多模态推理

GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（基础模型研究中心，自动化研究所，中国科学院）；ByteDance（字节跳动）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究所）；Xi’an Jiaotong University（西安交通大学）；Renmin University of China（中国人民大学）

Yufei Zhan, Ziheng Wu, Yousong Zhu, Rongkun Xue, Ruipu Luo, Zhenghao Chen, Can Zhang, Yifan Li, Zhentao He, Zheming Yang, Ming Tang, Minghui Qiu, Jinqiao Wang

备注：: Tech report

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

GThinker提出Cue-Rethinking推理模式和两阶段训练流程，并构建GThinker-11K数据集，在通用多模态推理基准上超越O4-mini并保持数学推理性能。

摘要：

尽管多模态推理取得了显著进展，领先的多模态大语言模型（MLLMs）在一般场景下以视觉为中心的多模态推理任务中仍然表现不佳。这一不足源于它们主要依赖基于逻辑和知识的慢速思考策略，这些策略虽然在数学和科学等领域有效，但在推理过程中未能有效整合视觉信息。因此，这些模型往往无法充分锚定视觉线索，导致在需要多种合理视觉解释和推断的任务中表现欠佳。为解决这一问题，我们提出了GThinker（General Thinker），一种新颖的推理型多模态大语言模型，在一般场景、数学和科学的多模态推理中表现出色。GThinker引入了Cue-Rethinking，一种灵活的推理模式，将推断锚定在视觉线索上，并迭代地重新解释这些线索以解决不一致之处。基于这一模式，我们进一步提出了两阶段训练流程，包括模式引导的冷启动和激励强化学习，旨在实现跨领域的多模态推理能力。此外，为支持训练，我们构建了GThinker-11K数据集，包含7K条高质量、迭代标注的推理路径和4K条精选的强化学习样本，填补了面向通用多模态推理的数据缺口。大量实验表明，GThinker在具有挑战性的综合多模态推理基准M$^3$CoT上达到了81.5%的成绩，超越了最新的O4-mini模型。在一般场景多模态推理基准上，它还取得了平均2.1%的提升，同时在数学推理方面与同类先进推理模型保持相当的性能。代码、模型和数据将很快在https://github.com/jefferyZhan/GThinker发布。

英文摘要：

Despite notable advancements in multimodal reasoning, leading Multimodal Large Language Models (MLLMs) still underperform on vision-centric multimodal reasoning tasks in general scenarios. This shortfall stems from their predominant reliance on logic- and knowledge-based slow thinking strategies, while effective for domains like math and science, fail to integrate visual information effectively during reasoning. Consequently, these models often fail to adequately ground visual cues, resulting in suboptimal performance in tasks that require multiple plausible visual interpretations and inferences. To address this, we present GThinker (General Thinker), a novel reasoning MLLM excelling in multimodal reasoning across general scenarios, mathematics, and science. GThinker introduces Cue-Rethinking, a flexible reasoning pattern that grounds inferences in visual cues and iteratively reinterprets these cues to resolve inconsistencies. Building on this pattern, we further propose a two-stage training pipeline, including pattern-guided cold start and incentive reinforcement learning, designed to enable multimodal reasoning capabilities across domains. Furthermore, to support the training, we construct GThinker-11K, comprising 7K high-quality, iteratively-annotated reasoning paths and 4K curated reinforcement learning samples, filling the data gap toward general multimodal reasoning. Extensive experiments demonstrate that GThinker achieves 81.5% on the challenging comprehensive multimodal reasoning benchmark M$^3$CoT, surpassing the latest O4-mini model. It also shows an average improvement of 2.1% on general scenario multimodal reasoning benchmarks, while maintaining on-par performance in mathematical reasoning compared to counterpart advanced reasoning models. The code, model, and data will be released soon at https://github.com/jefferyZhan/GThinker.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。