arXiv 2601.047772026-01-08cs.CVcs.AI

GeM-VG：迈向通用多图像视觉 grounding 的多模态大语言模型

GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models

Shurong Zheng, Yousong Zhu, Hongyin Zhao, Fan Yang, Yufei Zhan, Ming Tang, Jinqiao Wang

AI总结：

GeM-VG通过引入MG-Data-240K数据集和混合强化微调策略，提升多图像视觉 grounding 和通用多图像理解能力。

摘要：

多模态大语言模型（MLLMs）在单图像 grounding 和通用多图像理解方面展现了显著的进步。最近，一些方法开始解决多图像 grounding 问题。然而，它们受限于单目标定位和有限的实用任务类型，由于缺乏对通用 grounding 任务的统一建模。因此，我们提出了GeM-VG，一种能够进行通用多图像视觉 grounding 的MLLM。为了支持这一点，我们系统地分类和组织现有的多图像 grounding 任务，根据其对跨图像线索和推理的依赖程度进行分类，并引入MG-Data-240K数据集，解决现有数据集在目标数量和图像关系方面的局限性。为了解决稳健处理多样化多图像 grounding 任务的挑战，我们进一步提出了一种混合强化微调策略，该策略整合了链式推理（CoT）和直接回答，考虑了它们的互补优势。该策略采用一种R1-like算法，由精心设计的规则基于奖励引导，有效提升了模型的整体感知和推理能力。广泛的实验表明了我们模型的优越通用 grounding 能力。对于多图像 grounding，它在MIG-Bench和MC-Bench上分别比先前领先的MLLMs高出2.0%和9.7%。在单图像 grounding 中，它在ODINW上比基模型提高了9.1%。此外，我们的模型在通用多图像理解方面也保持了强大的能力。

英文摘要：

Multimodal Large Language Models (MLLMs) have demonstrated impressive progress in single-image grounding and general multi-image understanding. Recently, some methods begin to address multi-image grounding. However, they are constrained by single-target localization and limited types of practical tasks, due to the lack of unified modeling for generalized grounding tasks. Therefore, we propose GeM-VG, an MLLM capable of Generalized Multi-image Visual Grounding. To support this, we systematically categorize and organize existing multi-image grounding tasks according to their reliance of cross-image cues and reasoning, and introduce the MG-Data-240K dataset, addressing the limitations of existing datasets regarding target quantity and image relation. To tackle the challenges of robustly handling diverse multi-image grounding tasks, we further propose a hybrid reinforcement finetuning strategy that integrates chain-of-thought (CoT) reasoning and direct answering, considering their complementary strengths. This strategy adopts an R1-like algorithm guided by a carefully designed rule-based reward, effectively enhancing the model's overall perception and reasoning capabilities. Extensive experiments demonstrate the superior generalized grounding capabilities of our model. For multi-image grounding, it outperforms the previous leading MLLMs by 2.0% and 9.7% on MIG-Bench and MC-Bench, respectively. In single-image grounding, it achieves a 9.1% improvement over the base model on ODINW. Furthermore, our model retains strong capabilities in general multi-image understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。