Due to the significant advancement of Natural Language Processing and Computer Vision-based models, Visual Question Answering (VQA) systems are becoming more intelligent and advanced. However, they are still error-prone when dealing with relatively complex questions. Therefore, it is important to understand the behaviour of the VQA models before adopting their results. In this paper, we introduce an interpretability approach for VQA models by generating counterfactual images. Specifically, the generated image is supposed to have the minimal possible change to the original image and leads the VQA model to give a different answer. In addition, our approach ensures that the generated image is realistic. Since quantitative metrics cannot be employed to evaluate the interpretability of the model, we carried out a user study to assess different aspects of our approach. In addition to interpreting the result of VQA models on single images, the obtained results and the discussion provides an extensive explanation of VQA models' behaviour.
While Visual Question Answering (VQA) models continue to push the state-of-the-art forward, they largely remain black-boxes - failing to provide insight into how or why an answer is generated. In this ongoing work, we propose addressing this shortcoming by learning to generate counterfactual images for a VQA model - i.e. given a question-image pair, we wish to generate a new image such that i) the VQA model outputs a different answer, ii) the new image is minimally different from the original, and iii) the new image is realistic. Our hope is that providing such counterfactual examples allows u…
Nanyang Technological University（南洋理工大学）；Renmin University of China（中国人民大学）；Alibaba Group（阿里巴巴集团）
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce th…
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Hikvision Research Institute（海康威视研究院）
Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ide…
Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly capture. However, these properties can be estimated by utilizing cues from relative object motion and the dynamics introduced by collisions. In this paper, we introduce CRIPP-VQA, a new video question answering dataset for reasoning about the implicit physical properties of objects in a scene. CRIPP-VQA contains videos of objects in motion, annotated with questions that involve …
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering
The Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）；Columbia University（哥伦比亚大学）；Nanyang Technological University（南洋理工大学）
University of California, San Diego（加利福尼亚大学圣迭戈分校）；SRI International（SRI国际公司）
In the domain of Visual Question Answering (VQA), studies have shown improvement in users' mental model of the VQA system when they are exposed to examples of how these systems answer certain Image-Question (IQ) pairs. In this work, we show that showing controlled counterfactual image-question examples are more effective at improving the mental model of users as compared to simply showing random examples. We compare a generative approach and a retrieval-based approach to show counterfactual examples. We use recent advances in generative adversarial networks (GANs) to generate counterfactual im…
Technion - Israel Institute of Technology（以色列理工学院）
Deep learning algorithms have shown promising results in visual question answering (VQA) tasks, but a more careful look reveals that they often do not understand the rich signal they are being fed with. To understand and better measure the generalization capabilities of VQA systems, we look at their robustness to counterfactually augmented data. Our proposed augmentations are designed to make a focused intervention on a specific property of the question such that the answer changes. Using these augmentations, we propose a new robustness measure, Robustness to Augmented Data (RAD), which measur…
A$^2$Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering
Macau University of Science and Technology（澳门科技大学）；Wuhan University of Technology（武汉理工大学）；Nanjing Institute of Technology（南京工程学院）；National Tsing Hua University（国立清华大学）
Ask4VG: Risk-Aware Question Selection for Reducing Prior-Driven Answers in Medical VQA
School of Microelectronics, Tianjin University, Tianjin 300072, China（天津大学电子工程学院，天津 300072，中国）；DISI, University of Trento, Trento, Italy（特伦托大学DISI研究所，意大利特伦托）；School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China（天津大学电气与信息工程学院，天津 300072，中国）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
School of Computing；Artificial Intelligence Southwest Jiaotong University Chengdu, China；Artificial Intelligence Southwest Jiaotong University Chengdu, China z\ c\；Institute of Innovative Research Tokyo Institute of Technology Yokohama, Japan
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Cohere Labs（Cohere实验室）；Eisai Inc.（艾伯维公司）；Indian Institute of Science, Bangalore（班加罗尔印度科学研究院）
University of Adelaide（阿德莱德大学）；South China University of Technology（华南理工大学）；University of Malaya（马来亚大学）；Huawei（华为）
Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize. This is visible in the fact that they are vulnerable to learning coincidental correlations in the data rather than deeper relations between image content and ideas expressed in language. We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages, and an evaluation process that co-opts a well understood image-based metric to reflect the method's ability to reason. Measuring reasoning directly encourages generalizatio…
Learning Situation Hyper-Graphs for Video Question Answering
University of Central Florida（中佛罗里达大学）；Mayo Clinic（梅奥诊所）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Goethe University Frankfurt（法兰克福歌德大学）；Western Michigan University（西密歇根大学）
MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
University of Georgia（佐治亚大学）；Harvard Medical School（哈佛医学院）；Chungbuk National University（Chungbuk国立大学）；Chungnam National University Hospital（Chungnam国立大学医院）；National University of Singapore（新加坡国立大学）；New York University（纽约大学）；University of Sydney（悉尼大学）；New Jersey Institute of Technology（新泽西理工学院）
