对比区域引导：在无需训练的情况下改进视觉-语言模型的 grounding
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
AI总结：
本文提出无需训练的 Contrastive Region Guidance（CRG），通过对比有无视觉提示的模型输出来抵消先验偏差，从而增强开源 VLM 的区域 grounding，并在多类视觉-语言任务上取得显著提升。
摘要：
突出图像中特别相关的区域，可以引导视觉-语言模型（VLM）更密切地关注这些感兴趣区域，从而提升其在各类视觉-语言（VL）任务上的性能。例如，可以为 VLM 提供“视觉提示”，即使用边界框等视觉标记勾画出关键图像区域。然而，当前能够结合视觉引导的 VLM 要么是专有的且成本高昂，要么需要在包含视觉提示的精心整理数据上进行代价高昂的训练。我们提出了 Contrastive Region Guidance（CRG，对比区域引导），这是一种无需训练的引导方法，使开源 VLM 能够响应视觉提示。CRG 对模型在有视觉提示和无视觉提示情况下产生的输出进行对比，从而分解出模型在缺少产生正确答案所需信息时进行回答所暴露出的偏差（即模型的先验）。CRG 在多种 VL 任务中取得了显著改进：当提供区域标注时，CRG 在 ViP-Bench 上将绝对准确率最高提升 11.1%；ViP-Bench 是包含识别、数学和对象关系推理等六类基于区域任务的集合。我们还展示了 CRG 适用于空间推理，在 What'sUp 上取得 10% 的改进；也适用于组合泛化，在 SugarCrepe 的两个具有挑战性的划分上分别将准确率提升 11.5% 和 7.5%；此外还适用于生成图像的图文对齐，在 SeeTRUE 上最高提升 8.4 个 AUROC 点和 6.8 个 F1 点。当缺少参考区域时，CRG 允许我们在 RefCOCO/+/g 和 Flickr30K Entities 等指代表达理解与短语 grounding 基准中对候选区域重新排序，准确率平均提升 3.2%。我们的分析探索了 CRG 的替代掩码策略，量化了 CRG 的概率偏移，并评估了区域引导强度的作用，从经验上验证了 CRG 的设计选择。
英文摘要：
Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs) on various vision-language (VL) tasks by guiding the model to attend more closely to these regions of interest. For example, VLMs can be given a "visual prompt", where visual markers such as bounding boxes delineate key image regions. However, current VLMs that can incorporate visual guidance are either proprietary and expensive or require costly training on curated data that includes visual prompts. We introduce Contrastive Region Guidance (CRG), a training-free guidance method that enables open-source VLMs to respond to visual prompts. CRG contrasts model outputs produced with and without visual prompts, factoring out biases revealed by the model when answering without the information required to produce a correct answer (i.e., the model's prior). CRG achieves substantial improvements in a wide variety of VL tasks: When region annotations are provided, CRG increases absolute accuracy by up to 11.1% on ViP-Bench, a collection of six diverse region-based tasks such as recognition, math, and object relationship reasoning. We also show CRG's applicability to spatial reasoning, with 10% improvement on What'sUp, as well as to compositional generalization -- improving accuracy by 11.5% and 7.5% on two challenging splits from SugarCrepe -- and to image-text alignment for generated images, where we improve by up to 8.4 AUROC and 6.8 F1 points on SeeTRUE. When reference regions are absent, CRG allows us to re-rank proposed regions in referring expression comprehension and phrase grounding benchmarks like RefCOCO/+/g and Flickr30K Entities, with an average gain of 3.2% in accuracy. Our analysis explores alternative masking strategies for CRG, quantifies CRG's probability shift, and evaluates the role of region guidance strength, empirically validating CRG's design choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
