AI systems' ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA). However, most of them are opaque black boxes with limited explanatory capability. This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning while capturing the style of comprehensible human explanations. Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods with both automatic evaluation metrics and human evaluation metrics.
Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
AMAP, Alibaba Group（阿里云实验室，阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）
UC Berkeley EECS（加州大学伯克利分校电子工程与计算机科学系）；Sony Corp.（索尼公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality,…
Visual question answering by using information from multiple modalities has attracted more and more attention in recent years. However, it is a very challenging task, as the visual content and natural language have quite different statistical properties. In this work, we present a method called Adversarial Multimodal Network (AMN) to better understand video stories for question answering. In AMN, as inspired by generative adversarial networks, we propose to learn multimodal feature representations by finding a more coherent subspace for video clips and the corresponding texts (e.g., subtitles …
University of Maryland, College Park（马里兰大学帕克分校）；SRI International（SRI国际研究所）
The success of deep learning has enabled advances in multimodal tasks that require non-trivial fusion of multiple input domains. Although multimodal models have shown potential in many problems, their increased complexity makes them more vulnerable to attacks. A Backdoor (or Trojan) attack is a class of security vulnerability wherein an attacker embeds a malicious secret behavior into a network (e.g. targeted misclassification) that is activated when an attacker-specified trigger is added to an input. In this work, we show that multimodal networks are vulnerable to a novel type of attack that …
Carnegie Mellon University（卡内基梅隆大学）；Waseda University（早稻田大学）；Yahoo Japan Corporation（雅虎日本公司）
Question categorization and expert retrieval methods have been crucial for information organization and accessibility in community question & answering (CQA) platforms. Research in this area, however, has dealt with only the text modality. With the increasing multimodal nature of web content, we focus on extending these methods for CQA questions accompanied by images. Specifically, we leverage the success of representation learning for text and images in the visual question answering (VQA) domain, and adapt the underlying concept and architecture for automated category classification and exper…
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Mathematics and Computer Science, Faculty of Science, Alexandria University（亚历山大大学数学与计算机科学系）；Sheikh Shakhbout Medical City（谢赫·沙赫布OUT医疗城）
Sorbonne Universités, UPMC Univ Paris 06, CNRS, LIP6 UMR 7606（索邦大学，巴黎第六大学，法国国家科学研究中心，LIP6）；Heuritech；Conservatoire National des Arts et Métiers（法国国立工艺学院）
Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the comple…
Technical University of Darmstadt（达姆施塔特工业大学）；New York University（纽约大学）；University of Cambridge（剑桥大学）
Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based mode…
HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language
Silo AI；Ahmadu Bello University（艾哈迈德·贝罗大学）；Bayero University Kano（卡诺贝罗大学）；Microsoft（微软）；Charles University（查理大学）；Thapar University（塔帕尔大学）；Idiap Research Institute（Idiap研究所）；ETH Zurich（苏黎世联邦理工学院）
We present MMFT-BERT(MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformer-based fusion method to fuse them together. Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an…
Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-training
Shenzhen Institute of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）；University of Chinese Academy of Science（中国科学院大学）；Technical University of Munich（慕尼黑工业大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；Southern Medical University（南方医科大学）
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
Mercer Mettl（默瑟梅特尔）；IGDTUW（英迪拉·甘地德里技术大学（女子））；Google Research（谷歌研究院）；Indian Institute of Technology Guwahati（古瓦哈蒂印度理工学院）；University of Pennsylvania（宾夕法尼亚大学）
We present a new pre-training method, Multimodal Inverse Cloze Task, for Knowledge-based Visual Question Answering about named Entities (KVQAE). KVQAE is a recently introduced task that consists in answering questions about named entities grounded in a visual context using a Knowledge Base. Therefore, the interaction between the modalities is paramount to retrieve information and must be captured with complex fusion models. As these models require a lot of training data, we design this pre-training task from existing work in textual Question Answering. It consists in considering a sentence as …
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Heuritech（赫瑞泰克公司）；Conservatoire National des Arts et Métiers（法国国立工艺学院）
Multimodal attentional networks are currently state-of-the-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks. In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question…
Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）
This paper presents a new model for the task of scene text visual question answering, in which questions about a given image can only be answered by reading and understanding scene text that is present in it. The proposed model is based on an attention mechanism that attends to multi-modal features conditioned to the question, allowing it to reason jointly about the textual and visual modalities in the scene. The output weights of this attention module over the grid of multi-modal spatial features are interpreted as the probability that a certain spatial location of the image contains the answ…
MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR（澳门科学技术大学计算机科学与工程学院）；Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology（湖北省交通物联网重点实验室，武汉理工大学）
PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Data Science & Artificial Intelligence Research Institute, China Unicom（中国unicom数据科学与人工智能研究院）；Unicom Data Intelligence, China Unicom（中国unicom数据智能）
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization
Florida Institute of Technology（佛罗里达理工学院）；FPT University（FPT大学）；Uppsala University（乌普萨拉大学）；Knovel Engineering Lab（诺弗尔工程实验室）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Université de Montréal（蒙特利尔大学）；University of Adelaide（阿德莱德大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）
Knowledge-based visual question answering requires the ability of associating external knowledge for open-ended cross-modal scene understanding. One limitation of existing solutions is that they capture relevant knowledge from text-only knowledge bases, which merely contain facts expressed by first-order predicates or language descriptions while lacking complex but indispensable multimodal knowledge for visual understanding. How to construct vision-relevant and explainable multimodal knowledge for the VQA scenario has been less studied. In this paper, we propose MuKEA to represent multimodal k…
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence（人工智能交叉学科项目）；Department of Intelligence and Information（智能与信息系）；Samsung Advanced Institute of Technology（三星先进技术研究院）
MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
School of Computing and Augmented Intelligence（计算与增强智能学院）；Arizona State University（亚利桑那州立大学）；Department of Computer Science（计算机科学系）；International Institute of Information Technology（国际信息科技研究所）
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；NiuTrans Research（NiuTrans研究院）；School of Software, Northeastern University（东北大学软件学院）
VietMEAgent: Culturally-Aware Few-Shot Multimodal Explanation for Vietnamese Visual Question Answering
Faculty Of Information Technology, VNU University of Engineering and Technology（信息科技学院，越南工程与技术大学）；IT-BT Convergence Technology Division, Vietnam-Korea Institute of Science and Technology（IT-BT融合技术部，越南-韩国科学技术院）；TADI Global Lab, TADI Global Company Limited（TADI全球实验室，TADI全球公司）；Faculty of Finance, Banking Academy of Vietnam（金融学院，越南银行学院）
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Heuritech（赫瑞泰克公司）；Conservatoire National des Arts et Métiers（法国国立工艺学院）
Multimodal representation learning is gaining more and more interest within the deep learning community. While bilinear models provide an interesting framework to find subtle combination of modalities, their number of parameters grows quadratically with the input dimensions, making their practical implementation within classical deep learning pipelines challenging. In this paper, we introduce BLOCK, a new multimodal fusion based on the block-superdiagonal tensor decomposition. It leverages the notion of block-term ranks, which generalizes both concepts of rank and mode ranks for tensors, alrea…
