它们相同吗？探索多模态大语言模型的视觉对应缺陷
Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs
AI总结：
本研究揭示多模态大语言模型在视觉匹配上存在系统性缺陷，构建了MMVM基准和SFT数据集，并提出对比模型CoLVA，其性能超越GPT-4o和现有开源模型。
摘要：
近年来，多模态大语言模型（MLLM）在视觉感知、推理能力和视觉-语言理解方面展现出强大的能力。然而，尽管在计算机视觉中寻找物体的视觉对应关系至关重要，MLLM的视觉匹配能力却很少被研究。我们的研究揭示，即使是最新的强模型如GPT-4o，其匹配能力仍存在系统性缺陷。为此，我们构建了一个多模态视觉匹配（MMVM）基准，以公平地评估超过30种不同的MLLM。MMVM基准由15个开源数据集和带有手动标注的互联网视频构建而成。我们根据所需线索和能力将MMVM基准的数据样本分为八个方面，以更全面地评估和分析当前的MLLM。此外，我们设计了一个自动标注流程来生成MMVM SFT数据集，包括220K条带有推理标注的视觉匹配数据。据我们所知，这是MLLM社区首个视觉对应数据集和基准。最后，我们提出了CoLVA，一种新颖的对比MLLM，包含两项新技术设计：带有对象级对比学习的细粒度视觉专家和指令增强策略。前者学习实例判别性标记，后者进一步提高指令遵循能力。CoLVA-InternVL2-4B在MMVM基准上达到了49.80%的总体准确率（OA），分别超过GPT-4o和最佳开源MLLM Qwen2VL-72B 7.15%和11.72%的OA。这些结果证明了我们MMVM SFT数据集和新颖技术设计的有效性。代码、基准、数据集和模型将公开发布。
英文摘要：
Recent advancements in multimodal large language models (MLLM) have shown a strong ability in visual perception, reasoning abilities, and vision-language understanding. However, the visual matching ability of MLLMs is rarely studied, despite finding the visual correspondence of objects is essential in computer vision. Our research reveals that the matching capabilities in recent MLLMs still exhibit systematic shortcomings, even with current strong MLLMs models, GPT-4o. In particular, we construct a Multimodal Visual Matching (MMVM) benchmark to fairly benchmark over 30 different MLLMs. The MMVM benchmark is built from 15 open-source datasets and Internet videos with manual annotation. We categorize the data samples of MMVM benchmark into eight aspects based on the required cues and capabilities to more comprehensively evaluate and analyze current MLLMs. In addition, we have designed an automatic annotation pipeline to generate the MMVM SFT dataset, including 220K visual matching data with reasoning annotation. To our knowledge, this is the first visual corresponding dataset and benchmark for the MLLM community. Finally, we present CoLVA, a novel contrastive MLLM with two novel technical designs: fine-grained vision expert with object-level contrastive learning and instruction augmentation strategy. The former learns instance discriminative tokens, while the latter further improves instruction following ability. CoLVA-InternVL2-4B achieves an overall accuracy (OA) of 49.80\% on the MMVM benchmark, surpassing GPT-4o and the best open-source MLLM, Qwen2VL-72B, by 7.15\% and 11.72\% OA, respectively. These results demonstrate the effectiveness of our MMVM SFT dataset and our novel technical designs. Code, benchmark, dataset, and models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
