arXiv 2402.132542024-06-12cs.CVcs.AIcs.CLcs.LG

CounterCurate：通过反事实示例增强物理与语义视觉-语言组合推理

CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples

University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Microsoft Research（微软研究院）

Jianrui Zhang, Mu Cai, Tengyang Xie, Yong Jae Lee

备注：: 15 pages, 6 figures, 12 tables, Project Page: https://countercurate.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

CounterCurate框架通过物理基础数据增强和语义反事实微调，显著提升了CLIP和LLaVA等模型的视觉-语言组合推理能力。

摘要：

我们提出CounterCurate，一个旨在全面提升对比式和生成式多模态模型视觉-语言组合推理能力的框架。具体而言，我们识别出两个尚未被充分探索的关键问题：对物理基础推理（计数和位置理解）的忽视，以及利用高性能文本和图像生成模型进行语义反事实微调的潜力。我们的工作开创性地提出了一种解决这些不足的方法。我们首先揭示了CLIP和LLaVA等多模态模型在物理基础组合推理中接近随机水平的表现。随后，我们使用基于物理基础的图像生成模型GLIGEN进行简单的数据增强以生成微调数据，从而带来了显著的性能提升：在我们新构建的Flickr30k-Positions基准上，CLIP和LLaVA分别提升了+33%和+37%。此外，我们利用高性能文本生成和图像生成模型（特别是GPT-4V和DALLE-3）的能力来构建具有挑战性的语义反事实样本，从而进一步增强了在SugarCrepe等基准上的组合推理能力，在这些基准上CounterCurate的表现优于GPT-4V。为促进未来研究，我们在https://countercurate.github.io上发布了我们的代码、数据集、基准和检查点。

英文摘要：

We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In particular, we identify two critical under-explored problems: the neglect of the physically grounded reasoning (counting and position understanding) and the potential of using highly capable text and image generation models for semantic counterfactual fine-tuning. Our work pioneers an approach that addresses these gaps. We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning. We then apply simple data augmentation using grounded image generation model GLIGEN to generate fine-tuning data, resulting in significant performance improvements: +33% and +37% for CLIP and LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, we exploit the capabilities of high-performing text generation and image generation models, specifically GPT-4V and DALLE-3, to curate challenging semantic counterfactuals, thereby further enhancing compositional reasoning capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms GPT-4V. To facilitate future research, we release our code, dataset, benchmark, and checkpoints at https://countercurate.github.io.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。