arXiv 2501.057672025-02-18cs.CLcs.AIcs.CV

Migician：揭示多模态大语言模型中自由形式多图像定位的魔力

Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

Beijing Jiaotong University（北京交通大学）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）

You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun

备注：: 21 pages, 8 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有多模态大语言模型在复杂多图像场景中定位精度不足的问题，提出首个自由形式多图像定位模型Migician，构建MGrounding-630k数据集和MIG-Bench基准，实验性能显著优于现有模型。

摘要：

近年来多模态大语言模型（MLLMs）的发展显著提升了其对单图像的细粒度感知能力和对多图像的综合理解能力。然而，现有MLLMs在复杂多图像场景中实现精确的定位仍面临挑战。为此，我们首先探索了一种将单图像定位与多图像理解相结合的思维链（CoT）框架。尽管该框架部分有效，但由于其非端到端的特性，仍不稳定且难以捕捉抽象视觉信息。因此，我们引入了Migician，这是首个能够在多图像间执行自由形式且精确的定位模型。为支持该模型，我们提出了MGrounding-630k数据集，该数据集包含从现有数据集衍生的多个多图像定位任务数据，以及新生成的自由形式定位指令遵循数据。此外，我们提出了MIG-Bench，一个专门用于评估多图像定位能力的综合基准。实验结果表明，我们的模型实现了显著优越的多图像定位能力，比现有最佳MLLMs高出24.94%，甚至超过了更大的70B模型。我们的代码、模型、数据集和基准已完全开源，地址为https://migician-vg.github.io/。

英文摘要：

The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images. However, existing MLLMs still face challenges in achieving precise grounding in complex multi-image scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework that integrates single-image grounding with multi-image comprehension. While partially effective, it remains unstable and struggles to capture abstract visual information due to its non-end-to-end nature. Therefore, we introduce Migician, the first multi-image grounding model capable of performing free-form and accurate grounding across multiple images. To support this, we present the MGrounding-630k dataset, which comprises data for several multi-image grounding tasks derived from existing datasets, along with newly generated free-form grounding instruction-following data. Furthermore, we propose MIG-Bench, a comprehensive benchmark specifically designed for evaluating multi-image grounding capabilities. Experimental results demonstrate that our model achieves significantly superior multi-image grounding capabilities, outperforming the best existing MLLMs by 24.94% and even surpassing much larger 70B models. Our code, model, dataset, and benchmark are fully open-sourced at https://migician-vg.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。