迈向基于多层级对比学习的更忠实视觉问答自然语言解释
Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
AI总结：
针对VQA中自然语言解释不忠实的问题，提出多层级对比学习模型MCLE，通过语义、图像和实例级事实与反事实样本对齐特征，提升解释与答案及图像的一致性。
摘要：
视觉问答中的自然语言解释（VQA-NLE）旨在通过生成自然语言句子来解释模型的决策过程，以增加用户对黑盒系统的信任。现有的事后方法在获得合理解释方面已取得显著进展。然而，此类事后解释并不总是与人类逻辑推理一致，存在以下问题：1）演绎不满足性，生成的解释在逻辑上无法推导出答案；2）事实不一致性，模型在生成反事实解释时未考虑图像中的事实；3）语义扰动不敏感性，模型无法识别由微小扰动引起的语义变化。这些问题降低了模型生成解释的忠实性。为解决上述问题，我们提出了一种新颖的基于自监督多层级对比学习的自然语言解释模型（MCLE），用于VQA，该模型利用语义级、图像级和实例级的事实与反事实样本。MCLE提取判别性特征，并将解释的特征空间与视觉问答对齐，以生成更一致的解释。我们在两个VQA-NLE基准上进行了大量实验、消融分析和案例研究，以证明我们方法的有效性。
英文摘要：
Natural language explanation in visual question answer (VQA-NLE) aims to explain the decision-making process of models by generating natural language sentences to increase users' trust in the black-box systems. Existing post-hoc methods have achieved significant progress in obtaining a plausible explanation. However, such post-hoc explanations are not always aligned with human logical inference, suffering from the issues on: 1) Deductive unsatisfiability, the generated explanations do not logically lead to the answer; 2) Factual inconsistency, the model falsifies its counterfactual explanation for answers without considering the facts in images; and 3) Semantic perturbation insensitivity, the model can not recognize the semantic changes caused by small perturbations. These problems reduce the faithfulness of explanations generated by models. To address the above issues, we propose a novel self-supervised \textbf{M}ulti-level \textbf{C}ontrastive \textbf{L}earning based natural language \textbf{E}xplanation model (MCLE) for VQA with semantic-level, image-level, and instance-level factual and counterfactual samples. MCLE extracts discriminative features and aligns the feature spaces from explanations with visual question and answer to generate more consistent explanations. We conduct extensive experiments, ablation analysis, and case study to demonstrate the effectiveness of our method on two VQA-NLE benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
