了解“不”：一种数据驱动的方法用于增强CLIP中的否定意识
Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
AI总结：
本文提出NegationCLIP，通过生成包含否定的数据增强CLIP的否定意识，同时提出NegRefCOCOg基准用于评估多模态模型的否定理解能力。
摘要：
尽管CLIP通过连接视觉和语言显著提升了多模态理解，但无法把握否定——例如无法区分“停车”与“禁止停车”——带来了重大挑战。通过分析公共CLIP模型预训练数据，我们提出这种限制源于缺乏包含否定的数据。为解决此问题，我们引入了数据生成管道，利用大型语言模型（LLM）和多模态LLM生成包含否定的描述。通过使用我们生成的数据微调CLIP，我们开发了NegationCLIP，它在增强否定意识的同时保持了通用性。此外，为了全面评估否定理解能力，我们提出了NegRefCOCOg-基准，专门用于测试VLMs在句子中不同位置和表达方式下理解否定的能力。在各种CLIP架构上的实验验证了我们数据生成管道在增强CLIP准确感知否定能力方面的有效性。此外，NegationCLIP增强的否定意识在各种多模态任务中具有实际应用，通过文本到图像生成和提及图像分割中的性能提升得到了证明。
英文摘要：
While CLIP has significantly advanced multimodal understanding by bridging vision and language, the inability to grasp negation - such as failing to differentiate concepts like "parking" from "no parking" - poses substantial challenges. By analyzing the data used in the public CLIP model's pre-training, we posit this limitation stems from a lack of negation-inclusive data. To address this, we introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions. Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality. Moreover, to enable a comprehensive evaluation of negation understanding, we propose NegRefCOCOg-a benchmark tailored to test VLMs' ability to interpret negation across diverse expressions and positions within a sentence. Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately. Additionally, NegationCLIP's enhanced negation awareness has practical applications across various multimodal tasks, demonstrated by performance gains in text-to-image generation and referring image segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
