VITRIX-CLIPIN：通过指令编辑数据与长描述增强CLIP的细粒度视觉理解能力
VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
AI总结：
针对CLIP细粒度视觉理解不足的问题，本文提出CLIP-IN框架，结合指令编辑数据构建的难负样本对比学习与带旋转位置编码的长描述训练，在多类任务上取得性能提升，还可减少多模态大模型的视觉幻觉。
摘要：
尽管CLIP等视觉语言模型（VLM）在视觉与语言对齐方面取得了成功，但其在细节层面的细粒度视觉理解能力仍是核心挑战。我们提出了全新框架CLIP-IN，通过两项核心创新提升CLIP的细粒度感知能力。首先，我们将原本用于图像编辑的指令编辑数据集作为难负样本图像-文本对的独特来源，搭配对称难负样本对比损失，使模型能够有效区分细微的视觉-语义差异。其次，CLIP-IN引入长描述性字幕，采用旋转位置编码来捕捉标准CLIP通常遗漏的丰富语义上下文。实验表明，CLIP-IN在MMVP基准测试及各类细粒度视觉识别任务上取得了显著提升，同时未损害其在通用分类与检索任务上的鲁棒零样本性能。关键的是，将CLIP-IN的视觉表征集成到多模态大语言模型中，可显著减少视觉幻觉并增强推理能力。本研究凸显了将针对性的基于指令的对比学习与全面描述信息相结合，以提升视觉语言模型细粒度理解能力的巨大潜力。
英文摘要：
Despite the success of Vision-Language Models (VLMs) like CLIP in aligning vision and language, their proficiency in detailed, fine-grained visual comprehension remains a key challenge. We present CLIP-IN, a novel framework that bolsters CLIP's fine-grained perception through two core innovations. Firstly, we leverage instruction-editing datasets, originally designed for image manipulation, as a unique source of hard negative image-text pairs. Coupled with a symmetric hard negative contrastive loss, this enables the model to effectively distinguish subtle visual-semantic differences. Secondly, CLIP-IN incorporates long descriptive captions, utilizing rotary positional encodings to capture rich semantic context often missed by standard CLIP. Our experiments demonstrate that CLIP-IN achieves substantial gains on the MMVP benchmark and various fine-grained visual recognition tasks, without compromising robust zero-shot performance on broader classification and retrieval tasks. Critically, integrating CLIP-IN's visual representations into Multimodal Large Language Models significantly reduces visual hallucinations and enhances reasoning abilities. This work underscores the considerable potential of synergizing targeted, instruction-based contrastive learning with comprehensive descriptive information to elevate the fine-grained understanding of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
