MEGL：多模态解释引导学习
MEGL: Multimodal Explanation-Guided Learning
AI总结：
本文提出多模态解释引导学习框架MEGL，通过融合视觉与文本解释并引入显著性驱动文本定位、文本监督和分布一致性损失，在Object-ME和Action-ME数据集上同时提升图像分类准确性与解释质量。
摘要：
解释人工智能（AI）模型的决策过程对于解决其“黑箱”特性至关重要，尤其是在图像分类等任务中。传统的可解释AI（XAI）方法通常依赖单一模态的解释，要么是视觉解释，要么是文本解释，而这两种解释各有其固有局限。视觉解释能够突出关键区域，但往往缺乏推理依据；文本解释能够提供上下文，却缺少空间定位。此外，两类解释都可能不一致或不完整，从而限制了它们的可靠性。为应对这些挑战，我们提出了一种新颖的多模态解释引导学习（MEGL）框架，该框架同时利用视觉和文本解释来增强模型可解释性并提升分类性能。我们的显著性驱动文本定位（SDTG）方法将来自视觉解释的空间信息融入文本推理中，从而提供具有空间定位且上下文丰富的解释。此外，我们引入了对视觉解释的文本监督，使视觉解释与文本推理保持一致，即使在缺少真实视觉标注的情况下也能实现。视觉解释分布一致性损失通过将生成的视觉解释与数据集层面的模式对齐来进一步增强视觉一致性，使模型能够从不完整的多模态监督中有效学习。我们在两个新数据集Object-ME和Action-ME上验证了MEGL，用于带有多模态解释的图像分类任务。实验结果表明，MEGL在预测准确性和视觉与文本两个领域的解释质量上均优于先前方法。我们的代码将在论文被接收后公开。
英文摘要：
Explaining the decision-making processes of Artificial Intelligence (AI) models is crucial for addressing their "black box" nature, particularly in tasks like image classification. Traditional eXplainable AI (XAI) methods typically rely on unimodal explanations, either visual or textual, each with inherent limitations. Visual explanations highlight key regions but often lack rationale, while textual explanations provide context without spatial grounding. Further, both explanation types can be inconsistent or incomplete, limiting their reliability. To address these challenges, we propose a novel Multimodal Explanation-Guided Learning (MEGL) framework that leverages both visual and textual explanations to enhance model interpretability and improve classification performance. Our Saliency-Driven Textual Grounding (SDTG) approach integrates spatial information from visual explanations into textual rationales, providing spatially grounded and contextually rich explanations. Additionally, we introduce Textual Supervision on Visual Explanations to align visual explanations with textual rationales, even in cases where ground truth visual annotations are missing. A Visual Explanation Distribution Consistency loss further reinforces visual coherence by aligning the generated visual explanations with dataset-level patterns, enabling the model to effectively learn from incomplete multimodal supervision. We validate MEGL on two new datasets, Object-ME and Action-ME, for image classification with multimodal explanations. Experimental results demonstrate that MEGL outperforms previous approaches in prediction accuracy and explanation quality across both visual and textual domains. Our code will be made available upon the acceptance of the paper.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
