在VQA中测量忠实且合理的视觉定位
Measuring Faithful and Plausible Visual Grounding in VQA
AI总结：
本文提出FPVG指标，用于定量衡量VQA模型是否识别并真正依赖与问题相关的图像对象，从而评估其视觉定位的忠实性与合理性，并在多种VQA架构上进行了验证。
摘要：
视觉问答（VQA）系统中的视觉定位（VG）指标主要旨在衡量系统在推断给定问题答案时对图像相关部分的依赖程度。缺乏VG一直是当前最先进的VQA系统中的一个常见问题，其表现可能为过度依赖不相关的图像部分，或完全忽视视觉模态。尽管VQA模型的推理能力通常通过一些定性示例加以说明，但大多数系统并未对其VG属性进行定量评估。我们认为，一个易于计算的、能够有意义地衡量系统VG的准则，有助于弥补这一不足，并为模型评估和分析增加另一个有价值的维度。为此，我们提出了一种新的VG指标，该指标能够捕捉模型是否a）识别场景中与问题相关的对象，以及b）在生成答案时确实依赖相关对象所包含的信息，即其视觉定位是否既“忠实”又“合理”。我们的指标称为“忠实且合理的视觉定位”（FPVG），对于大多数VQA模型设计而言，该指标可以直接确定。我们详细描述了FPVG，并评估了涵盖多种VQA架构的若干参考系统。支持在GQA数据集上进行该指标计算的代码已在GitHub上提供。
英文摘要：
Metrics for Visual Grounding (VG) in Visual Question Answering (VQA) systems primarily aim to measure a system's reliance on relevant parts of the image when inferring an answer to the given question. Lack of VG has been a common problem among state-of-the-art VQA systems and can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. Although inference capabilities of VQA models are often illustrated by a few qualitative illustrations, most systems are not quantitatively assessed for their VG properties. We believe, an easily calculated criterion for meaningfully measuring a system's VG can help remedy this shortcoming, as well as add another valuable dimension to model evaluations and analysis. To this end, we propose a new VG metric that captures if a model a) identifies question-relevant objects in the scene, and b) actually relies on the information contained in the relevant objects when producing its answer, i.e., if its visual grounding is both "faithful" and "plausible". Our metric, called "Faithful and Plausible Visual Grounding" (FPVG), is straightforward to determine for most VQA model designs. We give a detailed description of FPVG and evaluate several reference systems spanning various VQA architectures. Code to support the metric calculations on the GQA data set is available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
