arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.04594cs.CVcs.AI
收藏

Img-Diff:面向多模态大语言模型的对比数据合成

Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

Sun Yat-Sen University(中山大学);Alibaba Group(阿里巴巴集团)

Qirui Jiao, Daoyuan Chen, Yilun Huang, Bolin Ding, Yaliang Li, Ying Shen

AI总结:

本文提出受对比学习启发的数据合成方法Img-Diff,通过生成相似图像对并利用差异区域与描述生成器构建高质量数据集,显著提升了多模态大模型在图像差异和视觉问答任务上的性能。

摘要:

高性能多模态大语言模型(MLLMs)严重依赖数据质量。为了推进MLLMs内的细粒度图像识别,我们引入了一种受对比学习和图像差异描述启发的新型数据合成方法。我们的核心思想是通过仔细观察相似图像中详细区域的物体差异,挑战模型辨别匹配和不同的元素。我们首先生成强调物体变化的相似图像对。随后,我们使用Difference Area Generator定位物体差异,接着使用Difference Captions Generator表达这些差异。这一过程产生了一个高质量的“物体替换”样本数据集,名为Img-Diff,由于其自动化特性,该数据集可根据需要进行扩展。我们利用这一生成的数据集微调最先进的(SOTA)MLLMs,如InternVL2,在各种图像差异和视觉问答任务中取得了显著改进。值得注意的是,经过训练的模型在MMVP基准上显著优于GPT-4V和Gemini等现有SOTA模型。此外,我们进行了全面评估以验证数据集的多样性、质量和鲁棒性,为这类对比数据集的合成提供了若干见解。我们发布代码和数据集,以鼓励对多模态数据合成及MLLMs图像理解基础能力的进一步研究。

英文摘要:

High-performance Multimodal Large Language Models (MLLMs) are heavily dependent on data quality. To advance fine-grained image recognition within MLLMs, we introduce a novel data synthesis method inspired by contrastive learning and image difference captioning. Our key idea involves challenging the model to discern both matching and distinct elements by scrutinizing object differences in detailed regions across similar images. We begin by generating pairs of similar images that emphasize object variations. Following this, we employ a Difference Area Generator to pinpoint object differences, and subsequently, a Difference Captions Generator to articulate these differences. This process results in a high-quality dataset of "object replacement" samples, termed Img-Diff, which can be scaled as needed due to its automated nature. We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs, such as InternVL2, achieving substantial improvements across various image difference and Visual Question Answering tasks. Notably, the trained models significantly outperform existing SOTA models like GPT-4V and Gemini on the MMVP benchmark. Additionally, we conduct comprehensive evaluations to validate the dataset's diversity, quality, and robustness, offering several insights into the synthesis of such contrastive datasets. We release our codes and dataset to encourage further research on multimodal data synthesis and MLLMs' fundamental capabilities for image understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑