Omni-NegCLIP：通过前层对比微调增强CLIP以实现全面否定理解
Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding
AI总结：
本文提出Omni-NegCLIP，通过修改CLIP的InfoNCE损失函数，改进其对存在性和缺失性否定表达的理解能力，实验显示在否定任务中性能提升显著，且不牺牲图像-文本检索能力。
摘要：
视觉-语言模型（VLMs）在多种多模态任务中表现出强大的能力。然而，最近的研究表明，如CLIP这样的VLM在理解否定表达方面表现不佳，这些表达在自然语言中很常见。在本工作中，我们提出Omni-NegCLIP，一种经过微调的CLIP模型，通过修改CLIP的原始InfoNCE对比损失，提高了CLIP对两种类型的否定表达的理解能力，即基于存在性的否定和基于缺失性的否定。具体而言，我们设计了一个基于存在的对比目标，将图像嵌入拉近其原始描述嵌入，同时推远相应的基于存在性的否定描述嵌入；另一个基于缺失性的对比目标，使图像嵌入与原始和基于缺失性的否定描述嵌入对齐，同时保持两种文本嵌入之间的语义区别。基于我们观察到CLIP文本编码器的前层变压器具有更强的否定文本学习能力，我们在每个训练步骤中使用结合的对比目标微调CLIP文本编码器的前层变压器。实验结果表明，与预训练的CLIP相比，Omni-NegCLIP在基于存在性的否定和基于缺失性的否定任务中性能分别提升高达52.65%和12.50%，不牺牲图像-文本检索的一般能力，甚至在检索任务中提升了19.62%。与先前工作相比，Omni-NegCLIP展示了更全面理解多种否定任务的能力。
英文摘要：
Vision-Language Models (VLMs) have demonstrated strong capabilities across a wide range of multimodal tasks. However, recent studies have shown that VLMs, such as CLIP, perform poorly in understanding negation expressions, which are common in natural language. In this work, we propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation, namely presence-based negation and absence-based negation, which correspond to negated expressions of objects that are actually present in an image and those that may plausibly exist in an image but are in fact absent, respectively, by modifying CLIP's original InfoNCE contrastive loss. Specifically, we design a presence-based contrastive objective that pulls image embeddings closer to their original caption embeddings while pushing them away from the corresponding presence-based negated caption embeddings, and an absence-based contrastive objective that aligns image embeddings with both original and absence-based negated caption embeddings while maintaining a semantic distinction between the two text embeddings. Based on our observation that the front transformer layers of CLIP text encoder have stronger learning ability for negated text than the later layers, we fine-tune the front transformer layers of the CLIP text encoder at each training step using the combined contrastive objective. Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively, without sacrificing general capability in image-text retrieval and even improving it by up to 19.62%. Compared with prior works, Omni-NegCLIP demonstrates a more comprehensive ability to understand multiple types of negation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
