视觉-语言预训练模型是否学习可组合的原始概念？
Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?
AI总结：
本文提出CompMap框架探究视觉-语言预训练模型是否自动涌现原始概念，发现其概念激活对细粒度识别和组合泛化有用但可解释性不足，揭示了现有模型局限及改进预训练目标的必要性。
摘要：
视觉-语言（VL）预训练模型在多模态推理和零样本识别任务上取得了令人瞩目的性能。许多此类VL模型是在互联网上未标注的图像-描述对上进行预训练的。本文研究原始概念（如颜色、形状或物体部件的属性）的表征是否会在这些预训练VL模型中自动涌现。我们提出一个两步框架——组合概念映射（CompMap）来探究这一问题。CompMap首先要求VL模型利用文本提示生成原始概念激活，然后学习构建一个组合模型，将原始概念激活（例如黑色尾巴或红色翅膀的可能性）映射到复合概念（例如红翅黑鹂）。我们表明，可以从真实标注的原始概念中可靠地学习组合模型。因此我们假设：如果原始概念确实在VL预训练模型中涌现，那么其原始概念激活可被用来学习一个与专家设计的组合模型相似的模型。我们提出一个定量指标来衡量相似程度，并将其称为可解释性指标。我们还测量使用原始概念激活和所学组合模型来预测复合概念时的分类准确率，并将其称为有用性指标。我们的研究表明，最先进的VL预训练模型学习到的原始概念对于CUB数据集上的细粒度视觉识别以及MIT-States数据集上的组合泛化任务非常有用。然而，我们在定性分析中观察到，所学组合模型的可解释性较低。我们的结果揭示了现有VL模型的局限性，以及设计鼓励获取原始概念的预训练目标的必要性。
英文摘要：
Vision-language (VL) pretrained models have achieved impressive performance on multimodal reasoning and zero-shot recognition tasks. Many of these VL models are pretrained on unlabeled image and caption pairs from the internet. In this paper, we study whether representations of primitive concepts--such as colors, shapes, or the attributes of object parts--emerge automatically within these pretrained VL models. We propose a two-step framework, Compositional Concept Mapping (CompMap), to investigate this. CompMap first asks a VL model to generate primitive concept activations with text prompts, and then learns to construct a composition model that maps the primitive concept activations (e.g. the likelihood of black tail or red wing) to composite concepts (e.g. a red-winged blackbird). We show that a composition model can be reliably learn from ground truth primitive concepts. We thus hypothesize that if primitive concepts indeed emerge in a VL pretrained model, its primitive concept activations can be used to learn a composition model similar to the one designed by experts. We propose a quantitative metric to measure the degree of similarity, and refer to the metric as the interpretability metric. We also measure the classification accuracy when using the primitive concept activations and the learned composition model to predict the composite concepts, and refer to it as the usefulness metric. Our study reveals that state-of-the-art VL pretrained models learn primitive concepts that are highly useful for fine-grained visual recognition on the CUB dataset, and compositional generalization tasks on the MIT-States dataset. However, we observe that the learned composition models have low interpretability in our qualitative analyses. Our results reveal the limitations of existing VL models, and the necessity of pretraining objectives that encourage the acquisition of primitive concepts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
