II-MMR：识别与改进视觉问答中的多模态多跳推理
II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering
AI总结：
本文提出II-MMR方法，通过答案预测引导CoT提示和知识三元组引导提示识别并改进VQA中的多模态多跳推理，在GQA和A-OKVQA基准上验证了其在零样本和微调设置下的有效性。
摘要：
视觉问答（VQA）通常涉及视觉与语言（V&L）领域的多种推理场景。然而，以往的大多数VQA研究仅关注评估模型的整体准确率，而未在不同推理情形下进行评估。此外，近期的一些研究观察到，传统的思维链（CoT）提示无法为VQA生成有效的推理，尤其是在需要多跳推理的复杂场景中。本文提出II-MMR，这是一种在VQA中识别并改进多模态多跳推理的新思路。具体而言，II-MMR接收一张图像和VQA问题，并利用两种新颖的语言提示找到推理路径以得出答案：(i) 答案预测引导的CoT提示，或(ii) 知识三元组引导的提示。随后，II-MMR通过估算回答问题所需的跳数及推理类型（即视觉或超视觉），分析该路径以识别当前VQA基准中的不同推理情形。在包含GQA和A-OKVQA的流行基准上，II-MMR观察到大多数VQA问题很容易回答，仅需“单跳”推理，而仅有少数问题需要“多跳”推理。此外，尽管近期的V&L模型即使使用传统CoT方法也难以应对此类复杂的多跳推理问题，但II-MMR在零样本和微调设置下的所有推理情形中均展现了其有效性。
英文摘要：
Visual Question Answering (VQA) often involves diverse reasoning scenarios across Vision and Language (V&L). Most prior VQA studies, however, have merely focused on assessing the model's overall accuracy without evaluating it on different reasoning cases. Furthermore, some recent works observe that conventional Chain-of-Thought (CoT) prompting fails to generate effective reasoning for VQA, especially for complex scenarios requiring multi-hop reasoning. In this paper, we propose II-MMR, a novel idea to identify and improve multi-modal multi-hop reasoning in VQA. In specific, II-MMR takes a VQA question with an image and finds a reasoning path to reach its answer using two novel language promptings: (i) answer prediction-guided CoT prompt, or (ii) knowledge triplet-guided prompt. II-MMR then analyzes this path to identify different reasoning cases in current VQA benchmarks by estimating how many hops and what types (i.e., visual or beyond-visual) of reasoning are required to answer the question. On popular benchmarks including GQA and A-OKVQA, II-MMR observes that most of their VQA questions are easy to answer, simply demanding "single-hop" reasoning, whereas only a few questions require "multi-hop" reasoning. Moreover, while the recent V&L model struggles with such complex multi-hop reasoning questions even using the traditional CoT method, II-MMR shows its effectiveness across all reasoning cases in both zero-shot and fine-tuning settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
