Deep models that are both effective and explainable are desirable in many settings; prior explainable models have been unimodal, offering either image-based visualization of attention weights or text-based generation of post-hoc justifications. We propose a multimodal approach to explanation, and argue that the two modalities provide complementary explanatory strengths. We collect two new datasets to define and evaluate this task, and propose a novel model which can provide joint textual rationale generation and attention visualization. Our datasets define visual and textual justifications of a classification decision for activity recognition tasks (ACT-X) and for visual question answering tasks (VQA-X). We quantitatively show that training with the textual explanations not only yields better textual justification models, but also better localizes the evidence that supports the decision. We also qualitatively show cases where visual explanation is more insightful than textual explanation, and vice versa, supporting our thesis that multimodal explanation models offer significant benefits over unimodal approaches.
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
School of Automation Science and Engineering, Xi’an Jiaotong University（自动化科学与工程学院，西安交通大学）；Interdisciplinary Graduate Programme, Nanyang Technological University（跨学科研究生项目，南洋理工大学）；Lenovo Research, Lenovo（联想研究院，联想）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
Evaluating Multimodal Large Language Models on Educational Textbook Question Answering
Faculty of Computing and Information Technology（计算与信息科技学院）；King Abdulaziz University（阿卜杜勒阿齐兹大学）；FAST School of Computing（快速计算学院）；National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）；School of Computing（计算学院）；Macquarie University（麦考瑞大学）；Center of Research Excellence in AI and Data Science（人工智能与数据科学卓越研究中心）
Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
AI Graduate Program, University of the Philippines（菲律宾大学人工智能研究生项目）；EEEI, University of the Philippines（菲律宾大学电子工程与信息学院）；Samsung R&D Institute Philippines（三星菲律宾研发研究所）
ExAID: A Multimodal Explanation Framework for Computer-Aided Diagnosis of Skin Lesions
German Research Center for Artificial Intelligence (DFKI) GmbH（德国人工智能研究中心）；Technical University Kaiserslautern（凯泽斯劳滕工业大学）；University Hospital Münster（明斯特大学医院）；University Hospital of Düsseldorf（杜塞尔多夫大学医院）；School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST)（国立科技大学电气工程与计算机科学学院）；Deep Learning Laboratory, National Center of Artificial Intelligence（国家人工智能中心深度学习实验室）
