通过生成反事实集合实现CLIP组合性推理的视觉飞跃
A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
AI总结：
本文提出基于块的扩散方法自动生成反事实图文数据集，并设计区分集合间与集合内样本的损失函数，以少量训练数据显著提升视觉-语言模型的组合性推理性能，在多个基准上取得最优结果。
摘要：
视觉-语言模型（VLMs）往往因缺乏高质量图文数据而在组合性推理方面表现不佳。为应对这一挑战，我们提出一种新颖的基于块的扩散方法，无需人工标注即可自动生成反事实数据集。我们的方法利用大型语言模型识别实体及其空间关系，随后独立生成图像块，将其作为“拼图块”按照指定的组合规则连贯排列。该过程能够生成多样化、高保真且具有精确可控变化的反事实图文对。此外，我们引入了一种专门的损失函数，用于区分集合间样本与集合内样本，从而提升训练效率并减少对负样本的需求。实验表明，使用我们的反事实数据集对视觉-语言模型进行微调可显著提升视觉推理性能。我们的方法在多个基准测试上取得了最先进的结果，同时使用的训练数据远少于现有方法。
英文摘要：
Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data. To tackle this challenge, we propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation. Our method utilizes large language models to identify entities and their spatial relationships. It then independently generates image blocks as "puzzle pieces" coherently arranged according to specified compositional rules. This process creates diverse, high-fidelity counterfactual image-text pairs with precisely controlled variations. In addition, we introduce a specialized loss function that differentiates inter-set from intra-set samples, enhancing training efficiency and reducing the need for negative samples. Experiments demonstrate that fine-tuning VLMs with our counterfactual datasets significantly improves visual reasoning performance. Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
