高性能多模态大语言模型(MLLMs)严重依赖数据质量。为了推进MLLMs内的细粒度图像识别,我们引入了一种受对比学习和图像差异描述启发的新型数据合成方法。我们的核心思想是通过仔细观察相似图像中详细区域的物体差异,挑战模型辨别匹配和不同的元素。我们首先生成强调物体变化的相似图像对。随后,我们使用Difference Area Generator定位物体差异,接着使用Difference Captions Generator表达这些差异。这一过程产生了一个高质量的“物体替换”样本数据集,名为Img-Diff,由于其自动化特性,该数据集可根据需要进行扩展。我们利用这一生成的数据集微调最先进的(SOTA)MLLMs,如InternVL2,在各种图像差异和视觉问答任务中取得了显著改进。值得注意的是,经过训练的模型在MMVP基准上显著优于GPT-4V和Gemini等现有SOTA模型。此外,我们进行了全面评估以验证数据集的多样性、质量和鲁棒性,为这类对比数据集的合成提供了若干见解。我们发布代码和数据集,以鼓励对多模态数据合成及MLLMs图像理解基础能力的进一步研究。
英文摘要:
High-performance Multimodal Large Language Models (MLLMs) are heavily dependent on data quality. To advance fine-grained image recognition within MLLMs, we introduce a novel data synthesis method inspired by contrastive learning and image difference captioning. Our key idea involves challenging the model to discern both matching and distinct elements by scrutinizing object differences in detailed regions across similar images. We begin by generating pairs of similar images that emphasize object variations. Following this, we employ a Difference Area Generator to pinpoint object differences, and subsequently, a Difference Captions Generator to articulate these differences. This process results in a high-quality dataset of "object replacement" samples, termed Img-Diff, which can be scaled as needed due to its automated nature. We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs, such as InternVL2, achieving substantial improvements across various image difference and Visual Question Answering tasks. Notably, the trained models significantly outperform existing SOTA models like GPT-4V and Gemini on the MMVP benchmark. Additionally, we conduct comprehensive evaluations to validate the dataset's diversity, quality, and robustness, offering several insights into the synthesis of such contrastive datasets. We release our codes and dataset to encourage further research on multimodal data synthesis and MLLMs' fundamental capabilities for image understanding.
Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);School of Computer Science and Tech., University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Information Engineering, CAS(信息工程研究所,中国科学院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院)
Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models
School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);Zhongguancun Academy(中关村学院);Southeast Academy of Information Technology, Beijing Institute of Technology(信息技术东南学院,北京理工大学);Tsinghua University(清华大学)
Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Princeton Neuroscience Institute(普林斯顿神经科学研究所);Medical AI Research Center (MedARC)(医学人工智能研究中心(MedARC));Ecole Normale Supérieure, PSL University(巴黎高等师范学院,PSL大学);University of Toronto(多伦多大学);Hebrew University of Jerusalem(希伯来大学);EleutherAI;Stability AI
A Cross Branch Fusion-Based Contrastive Learning Framework for Point Cloud Self-supervised Learning
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Tsinghua University(清华大学);Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB(弗劳恩霍夫光电、系统技术和图像处理研究所)
DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation
Westlake University(西湖大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);National University of Singapore(新加坡国立大学);Hupan Lab, Zhejiang Province(浙江省湖畔实验室);AI Lab, Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心人工智能实验室)