基于知识的反事实查询用于视觉问答
Knowledge-Based Counterfactual Queries for Visual Question Answering
AI总结：
本文提出利用结构化知识库对VQA输入问题进行词级反事实扰动，通过分析模型在不同对抗情形下的响应，揭示其决策偏差与推理模式，从而以模型无关的方式提升可解释性与鲁棒性分析。
摘要：
视觉问答（VQA）是一项结合视觉与语言的热门任务，文献中已有大量相关实现。尽管已有一些尝试探讨VQA模型的可解释性与鲁棒性问题，但其中很少有工作以模型无关的方式利用反事实来探查这些挑战。在本工作中，我们提出了一种系统方法，通过反事实扰动来解释VQA模型的行为并研究其鲁棒性。为此，我们利用结构化知识库，针对语言模态执行确定性的、最优的且可控的词级替换，然后评估模型在此类反事实输入下的响应。最后，我们基于反事实响应定性地提取局部和全局解释，这些解释最终被证明对理解VQA模型行为具有启发性。通过对输入问题中不同词性执行多种扰动类型，我们通过比较模型在不同对抗情形下的响应，获得了对其推理过程的洞察。总体而言，我们揭示了模型决策过程中可能存在的偏差，以及预期和非预期的模式，这些模式在定量和定性上均影响其性能，正如我们的分析所示。
英文摘要：
Visual Question Answering (VQA) has been a popular task that combines vision and language, with numerous relevant implementations in literature. Even though there are some attempts that approach explainability and robustness issues in VQA models, very few of them employ counterfactuals as a means of probing such challenges in a model-agnostic way. In this work, we propose a systematic method for explaining the behavior and investigating the robustness of VQA models through counterfactual perturbations. For this reason, we exploit structured knowledge bases to perform deterministic, optimal and controllable word-level replacements targeting the linguistic modality, and we then evaluate the model's response against such counterfactual inputs. Finally, we qualitatively extract local and global explanations based on counterfactual responses, which are ultimately proven insightful towards interpreting VQA model behaviors. By performing a variety of perturbation types, targeting different parts of speech of the input question, we gain insights to the reasoning of the model, through the comparison of its responses in different adversarial circumstances. Overall, we reveal possible biases in the decision-making process of the model, as well as expected and unexpected patterns, which impact its performance quantitatively and qualitatively, as indicated by our analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
