arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.07796cs.CLcs.CV
收藏

CREPE:视觉-语言基础模型能否进行组合性推理?

CREPE: Can Vision-Language Foundation Models Reason Compositionally?

Stanford University(斯坦福大学);Cornell University(康奈尔大学);University of Pennsylvania(宾夕法尼亚大学);University of Washington(华盛顿大学)

Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, Ranjay Krishna

AI总结:

本文提出组合性评估基准CREPE,从系统性和生产力两方面测试视觉-语言模型,发现7种架构在组合推理上均表现挣扎,性能随复杂度增加而显著下降。

摘要:

人类视觉和自然语言共同的一个基本特征是其组合性。然而,尽管大规模视觉和语言预训练带来了性能提升,我们发现:在海量数据集上使用4种算法训练的7种架构在组合性方面表现挣扎。为得出此结论,我们引入了一个新的组合性评估基准CREPE,它衡量认知科学文献中确定的组合性的两个重要方面:系统性和生产力。为衡量系统性,CREPE包含一个含超过37万个图文对的测试数据集和三个不同的已见-未见划分。这三个划分旨在测试在三个流行训练数据集上训练的模型:CC-12M、YFCC-15M和LAION-400M。我们还为部分图文对生成了32.5万、31.6万和30.9万条困难负样本描述。为测试生产力,CREPE包含1.7万个具有九种不同复杂度的图文对,以及18.3万条带有原子、交换和否定干扰项的困难负样本描述。这些数据集通过重新利用Visual Genome场景图和区域描述,并应用手工模板和GPT-3生成。对于系统性,我们发现当新组合主导检索集时,模型性能持续下降,Recall@1最多下降12%。对于生产力,随着复杂度增加,模型的检索成功率下降,在高复杂度时经常接近随机概率。无论模型和训练数据集大小如何,这些结果均成立。

英文摘要:

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, we find that: across 7 architectures trained with 4 algorithms on massive datasets, they struggle at compositionality. To arrive at this conclusion, we introduce a new compositionality evaluation benchmark, CREPE, which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. To measure systematicity, CREPE consists of a test dataset containing over $370K$ image-text pairs and three different seen-unseen splits. The three splits are designed to test models trained on three popular training datasets: CC-12M, YFCC-15M, and LAION-400M. We also generate $325K$, $316K$, and $309K$ hard negative captions for a subset of the pairs. To test productivity, CREPE contains $17K$ image-text pairs with nine different complexities plus $183K$ hard negative captions with atomic, swapping and negation foils. The datasets are generated by repurposing the Visual Genome scene graphs and region descriptions and applying handcrafted templates and GPT-3. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set, with Recall@1 dropping by up to $12\%$. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity. These results hold regardless of model and training dataset size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑