Deep learning methods have proven extremely effective at performing a variety of medical image analysis tasks. With their potential use in clinical routine, their lack of transparency has however been one of their few weak points, raising concerns regarding their behavior and failure modes. While most research to infer model behavior has focused on indirect strategies that estimate prediction uncertainties and visualize model support in the input image space, the ability to explicitly query a prediction model regarding its image content offers a more direct way to determine the behavior of trained models. To this end, we present a novel Visual Question Answering approach that allows an image to be queried by means of a written question. Experiments on a variety of medical and natural image datasets show that by fusing image and question features in a novel way, the proposed approach achieves an equal or higher accuracy compared to current methods.
Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Amsterdam（阿姆斯特丹大学）；Digital Futures（数字未来机构）；Helmholtz AI（亥姆霍兹人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；MPI for Intelligent Systems, Tübingen（蒂宾根马克斯·普朗克智能系统研究所）
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
Transfer learning is an important step to extract meaningful features and overcome the data limitation in the medical Visual Question Answering (VQA) task. However, most of the existing medical VQA methods rely on external data for transfer learning, while the meta-data within the dataset is not fully utilized. In this paper, we present a new multiple meta-model quantifying method that effectively learns meta-annotation and leverages meaningful features to the medical VQA task. Our proposed method is designed to increase meta-data by auto-annotation, deal with noisy labels, and output meta-mod…
University of California, Los Angeles（加州大学洛杉矶分校）；Amazon（亚马逊）
Outside-knowledge visual question answering (OK-VQA) requires the agent to comprehend the image, make use of relevant knowledge from the entire web, and digest all the information to answer the question. Most previous works address the problem by first fusing the image and question in the multi-modal space, which is inflexible for further fusion with a vast amount of external knowledge. In this paper, we call for a paradigm shift for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the natural l…
IBM Research China（IBM中国研究院）；IBM Research, Almaden Research Center（IBM研究院阿尔马登研究中心）
Visual Question Answering (VQA) becomes one of the most active research problems in the medical imaging domain. A well-known VQA challenge is the intrinsic diversity between the image and text modalities, and in the medical VQA task, there is another critical problem relying on the limited size of labelled image-question-answer data. In this study we propose an encoder-decoder framework that leverages the image-text joint representation learned from large-scaled medical image-caption data and adapted to the small-sized medical VQA task. The encoder embeds across the image-text dual modalities …
AIOZ Pte Ltd（AIOZ私人有限公司）；University of Liverpool（利物浦大学）
Traditional approaches for Visual Question Answering (VQA) require large amount of labeled data for training. Unfortunately, such large scale data is usually not available for medical domain. In this paper, we propose a novel medical VQA framework that overcomes the labeled data limitation. The proposed framework explores the use of the unsupervised Denoising Auto-Encoder (DAE) and the supervised Meta-Learning. The advantage of DAE is to leverage the large amount of unlabeled images while the advantage of Meta-Learning is to learn meta-weights that quickly adapt to VQA problem with limited lab…
This paper introduces MedMCQA, a new large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. More than 194k high-quality AIIMS \& NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity. Each sample contains a question, correct answer(s), and other options which requires a deeper language understanding as it tests the 10+ reasoning abilities of a model across a wide range of medical subjects \& topics. A detailed explan…
Prompt-based Personalized Federated Learning for Medical Visual Question Answering
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究生院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
Structure Causal Models and LLMs Integration in Medical Visual Question Answering
School of Microelectronics, Tianjin University（天津大学微电子学院）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
University of Science and Technology Beijing（北京科技大学）；Key Laboratory of Knowledge Automation for Industrial Processes, Ministry of Education（教育部工业过程知识自动化重点实验室）；Shunde Innovation School, University of Science and Technology Beijing（北京科技大学顺德创新学院）
There is a key problem in the medical visual question answering task that how to effectively realize the feature fusion of language and medical images with limited datasets. In order to better utilize multi-scale information of medical images, previous methods directly embed the multi-stage visual feature maps as tokens of same size respectively and fuse them with text representation. However, this will cause the confusion of visual features at different stages. To this end, we propose a simple but powerful multi-stage feature fusion method, MF2-MVQA, which stage-wise fuses multi-level visual …
Visual Question Answering (VQA) models take an image and a natural-language question as input and infer the answer to the question. Recently, VQA systems in medical imaging have gained popularity thanks to potential advantages such as patient engagement and second opinions for clinicians. While most research efforts have been focused on improving architectures and overcoming data-related limitations, answer consistency has been overlooked even though it plays a critical role in establishing trustworthy models. In this work, we propose a novel loss function and corresponding training procedure …
Harbin Engineering University（哈尔滨工程大学）；Harbin Medical University（哈尔滨医科大学）
Medical Visual Question Answering (VQA) is a multi-modal challenging task widely considered by research communities of the computer vision and natural language processing. Since most current medical VQA models focus on visual content, ignoring the importance of text, this paper proposes a multi-view attention-based model(MuVAM) for medical visual question answering which integrates the high-level semantics of medical images on the basis of text description. Firstly, different methods are utilized to extract the features of the image and the question for the two modalities of vision and text. S…
The Hong Kong Polytechnic University（香港理工大学）；West China Hospital of Sichuan University（四川大学华西医院）；Sichuan Academy of Medical Sciences（四川省医学科学院）；Sichuan Provincial People's Hospital（四川省人民医院）
Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the developmen…
Visual Question Answering in Medical domain (VQA-Med) plays an important role in providing medical assistance to the end-users. These users are expected to raise either a straightforward question with a Yes/No answer or a challenging question that requires a detailed and descriptive answer. The existing techniques in VQA-Med fail to distinguish between the different question types sometimes complicates the simpler problems, or over-simplifies the complicated ones. It is certainly true that for different question types, several distinct systems can lead to confusion and discomfort for the end-u…
Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
School of Artificial Intelligence, Tiangong University（天津工业大学人工智能学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Linying Medical Technology (Shenzhen) Co., Ltd.（临影医疗科技（深圳）有限公司）
Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering
Tandon School of Engineering, New York University（纽约大学工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
IBM Research AI, Almaden Research Center（IBM研究AI，阿尔马登研究中心）
In this working notes paper, we describe IBM Research AI (Almaden) team's participation in the ImageCLEF 2019 VQA-Med competition. The challenge consists of four question-answering tasks based on radiology images. The diversity of imaging modalities, organs and disease types combined with a small imbalanced training set made this a highly complex problem. To overcome these difficulties, we implemented a modular pipeline architecture that utilized transfer learning and multi-task learning. Our findings led to the development of a novel model called Supporting Facts Network (SFN). The main idea …
Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering
The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；RIKEN（理化学研究所）；The University of Tokyo（东京大学）；National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；National Cancer Center Research Institute（国立癌症中心研究所）
Contrastive Language--Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image--text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. This work evaluates the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). To this end, we present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments are conducted on two MedVQA benchmark datasets and investigate two MedVQA met…
Saliency Guided Longitudinal Medical Visual Question Answering
Dept. of Computer Science and Engineering University of California, San Diego（计算机科学与工程系，加州大学圣地亚哥分校）；Dept. of Radiology and Biomedical Imaging Yale University（放射学与生物医学成像系，耶鲁大学）
Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
School of Computing；Artificial Intelligence Southwest Jiaotong University Chengdu, China；Artificial Intelligence Southwest Jiaotong University Chengdu, China z\ c\；Institute of Innovative Research Tokyo Institute of Technology Yokohama, Japan
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
Faculty Of Information Technology VNU University of Engineering（信息科技学院越南工程大学）；IT-BT Convergence Technology Division Vietnam-Korea Institute of Science（IT-BT融合技术部门越南-韩国科学技术院）；TADI Global Lab TADI Global Company Limited（TADI全球实验室TADI全球有限公司）；Faculty of Finance Banking Academy of Vietnam（金融学院越南银行学院）
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Laboratory（湖畔实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Department of Radiology, The Affiliated Yangming Hospital of Ningbo University（宁波大学附属阳明医院放射科）；Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院，浙江大学）；Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University（清华长庚医院肝胆胰中心，清华大学临床医学院，清华医学，清华大学）；School of Software, Tsinghua University（清华大学软件学院）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
Johns Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia（意大利理工学院生物分子纳米技术中心）
