论视觉定位在VQA中的作用
On the Role of Visual Grounding in VQA
AI总结：
本研究提出VGR理论框架，形式化视觉定位在VQA中的作用，揭示OOD测试中捷径学习问题，并提出强调VG需求的OOD测试构建方法以提升性能。
摘要：
视觉定位（Visual Grounding, VG）在VQA中指的是模型基于问题相关的图像区域来推断答案的倾向。从概念上讲，VG被认为是VQA任务的公理性要求。然而，在实践中，基于深度神经网络（DNN）的VQA模型因通过捷径（SC）学习绕过VG而闻名，且在标准基准测试中并未遭受明显的性能损失。为了揭示SC学习的影响，研究者提出了分布外（OOD）测试，这些测试通过低准确率暴露了VG的缺失。这些测试自此成为VG研究的核心，并作为各种关于VG对准确率影响的研究基础。然而，VG在VQA中的作用仍未得到完全理解，也尚未被正式形式化。在本工作中，我们试图通过在概念层面上形式化VG来阐明其在VQA中的作用。我们提出了一个名为“视觉定位推理”（Visually Grounded Reasoning, VGR）的新颖理论框架，该框架利用VG和推理的概念来描述理想OOD测试中的VQA推理。通过整合关于VG在VQA中作用的基本见解，VGR有助于揭示OOD测试中普遍存在的与VG相关的捷径利用现象，这解释了为何VG与OOD准确率之间的关系难以定义。最后，我们提出了一种创建OOD测试的方法，以适当地强调对VG的需求，并展示了如何在这些测试上提升性能。
英文摘要：
Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers based on question-relevant image regions. Conceptually, VG identifies as an axiomatic requirement of the VQA task. In practice, however, DNN-based VQA models are notorious for bypassing VG by way of shortcut (SC) learning without suffering obvious performance losses in standard benchmarks. To uncover the impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that expose a lack of VG with low accuracy. These tests have since been at the center of VG research and served as basis for various investigations into VG's impact on accuracy. However, the role of VG in VQA still remains not fully understood and has not yet been properly formalized. In this work, we seek to clarify VG's role in VQA by formalizing it on a conceptual level. We propose a novel theoretical framework called "Visually Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to describe VQA inference in ideal OOD testing. By consolidating fundamental insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC exploitation in OOD testing, which explains why the relationship between VG and OOD accuracy has been difficult to define. Finally, we propose an approach to create OOD tests that properly emphasize a requirement for VG, and show how to improve performance on them.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
