Q-GroundCAM：通过 GradCAM 量化视觉语言模型中的定位能力
Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
AI总结：
本文提出 Q-GroundCAM 指标套件，利用 GradCAM 激活量化评估 CLIP、BLIP 和 ALBEF 等预训练视觉语言模型的零样本定位能力，揭示模型规模、数据集大小与性能之间的权衡关系。
摘要：
视觉与语言模型（VLMs）持续在各种任务中展现出卓越的零样本（ZS）性能。然而，许多探究性研究揭示，即使是表现最佳的 VLMs 也难以捕捉组合式场景理解的各个方面，缺乏将语言短语在图像中正确定位和锚定的能力。近期 VLM 的进展包括扩大模型和数据集规模、增加额外的训练目标和监督级别，以及模型架构的变化。为了刻画 VLMs 的定位能力，例如短语定位、指代表达理解和关系理解，Pointing Game 已被用作具有边界框标注的数据集的评估指标。在本文中，我们引入了一套新颖的定量指标，利用 GradCAM 激活来严格评估预训练 VLMs（如 CLIP、BLIP 和 ALBEF）的定位能力。这些指标提供了一种可解释且可量化的方法，用于更详细地比较 VLMs 的零样本能力，并能够测量模型的定位不确定性。这一刻画揭示了模型规模、数据集大小与其性能之间的有趣权衡。
英文摘要：
Vision and Language Models (VLMs) continue to demonstrate remarkable zero-shot (ZS) performance across various tasks. However, many probing studies have revealed that even the best-performing VLMs struggle to capture aspects of compositional scene understanding, lacking the ability to properly ground and localize linguistic phrases in images. Recent VLM advancements include scaling up both model and dataset sizes, additional training objectives and levels of supervision, and variations in the model architectures. To characterize the grounding ability of VLMs, such as phrase grounding, referring expressions comprehension, and relationship understanding, Pointing Game has been used as an evaluation metric for datasets with bounding box annotations. In this paper, we introduce a novel suite of quantitative metrics that utilize GradCAM activations to rigorously evaluate the grounding capabilities of pre-trained VLMs like CLIP, BLIP, and ALBEF. These metrics offer an explainable and quantifiable approach for a more detailed comparison of the zero-shot capabilities of VLMs and enable measuring models' grounding uncertainty. This characterization reveals interesting tradeoffs between the size of the model, the dataset size, and their performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
