Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs to enable learning from multimodal data. Key areas we address include the exploration of medical vision-language datasets, in-depth analyses of architectures and pre-training strategies employed in recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges and propose future directions, including enhancing clinical validity and addressing patient privacy concerns. Overall, our review summarizes recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.
Visual Question Answering (VQA) models take an image and a natural-language question as input and infer the answer to the question. Recently, VQA systems in medical imaging have gained popularity thanks to potential advantages such as patient engagement and second opinions for clinicians. While most research efforts have been focused on improving architectures and overcoming data-related limitations, answer consistency has been overlooked even though it plays a critical role in establishing trustworthy models. In this work, we propose a novel loss function and corresponding training procedure …
Transfer learning is an important step to extract meaningful features and overcome the data limitation in the medical Visual Question Answering (VQA) task. However, most of the existing medical VQA methods rely on external data for transfer learning, while the meta-data within the dataset is not fully utilized. In this paper, we present a new multiple meta-model quantifying method that effectively learns meta-annotation and leverages meaningful features to the medical VQA task. Our proposed method is designed to increase meta-data by auto-annotation, deal with noisy labels, and output meta-mod…
AIOZ Pte Ltd（AIOZ私人有限公司）；University of Liverpool（利物浦大学）
Traditional approaches for Visual Question Answering (VQA) require large amount of labeled data for training. Unfortunately, such large scale data is usually not available for medical domain. In this paper, we propose a novel medical VQA framework that overcomes the labeled data limitation. The proposed framework explores the use of the unsupervised Denoising Auto-Encoder (DAE) and the supervised Meta-Learning. The advantage of DAE is to leverage the large amount of unlabeled images while the advantage of Meta-Learning is to learn meta-weights that quickly adapt to VQA problem with limited lab…
Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering
The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；RIKEN（理化学研究所）；The University of Tokyo（东京大学）；National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；National Cancer Center Research Institute（国立癌症中心研究所）
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
Prompt-based Personalized Federated Learning for Medical Visual Question Answering
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究生院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
Structure Causal Models and LLMs Integration in Medical Visual Question Answering
School of Microelectronics, Tianjin University（天津大学微电子学院）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
IBM Research China（IBM中国研究院）；IBM Research, Almaden Research Center（IBM研究院阿尔马登研究中心）
Visual Question Answering (VQA) becomes one of the most active research problems in the medical imaging domain. A well-known VQA challenge is the intrinsic diversity between the image and text modalities, and in the medical VQA task, there is another critical problem relying on the limited size of labelled image-question-answer data. In this study we propose an encoder-decoder framework that leverages the image-text joint representation learned from large-scaled medical image-caption data and adapted to the small-sized medical VQA task. The encoder embeds across the image-text dual modalities …
Harbin Engineering University（哈尔滨工程大学）；Harbin Medical University（哈尔滨医科大学）
Medical Visual Question Answering (VQA) is a multi-modal challenging task widely considered by research communities of the computer vision and natural language processing. Since most current medical VQA models focus on visual content, ignoring the importance of text, this paper proposes a multi-view attention-based model(MuVAM) for medical visual question answering which integrates the high-level semantics of medical images on the basis of text description. Firstly, different methods are utilized to extract the features of the image and the question for the two modalities of vision and text. S…
Visual Question Answering in Medical domain (VQA-Med) plays an important role in providing medical assistance to the end-users. These users are expected to raise either a straightforward question with a Yes/No answer or a challenging question that requires a detailed and descriptive answer. The existing techniques in VQA-Med fail to distinguish between the different question types sometimes complicates the simpler problems, or over-simplifies the complicated ones. It is certainly true that for different question types, several distinct systems can lead to confusion and discomfort for the end-u…
Deep learning methods have proven extremely effective at performing a variety of medical image analysis tasks. With their potential use in clinical routine, their lack of transparency has however been one of their few weak points, raising concerns regarding their behavior and failure modes. While most research to infer model behavior has focused on indirect strategies that estimate prediction uncertainties and visualize model support in the input image space, the ability to explicitly query a prediction model regarding its image content offers a more direct way to determine the behavior of tra…
University of Science and Technology Beijing（北京科技大学）；Key Laboratory of Knowledge Automation for Industrial Processes, Ministry of Education（教育部工业过程知识自动化重点实验室）；Shunde Innovation School, University of Science and Technology Beijing（北京科技大学顺德创新学院）
There is a key problem in the medical visual question answering task that how to effectively realize the feature fusion of language and medical images with limited datasets. In order to better utilize multi-scale information of medical images, previous methods directly embed the multi-stage visual feature maps as tokens of same size respectively and fuse them with text representation. However, this will cause the confusion of visual features at different stages. To this end, we propose a simple but powerful multi-stage feature fusion method, MF2-MVQA, which stage-wise fuses multi-level visual …
The Hong Kong Polytechnic University（香港理工大学）；West China Hospital of Sichuan University（四川大学华西医院）；Sichuan Academy of Medical Sciences（四川省医学科学院）；Sichuan Provincial People's Hospital（四川省人民医院）
Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the developmen…
Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
School of Artificial Intelligence, Tiangong University（天津工业大学人工智能学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Linying Medical Technology (Shenzhen) Co., Ltd.（临影医疗科技（深圳）有限公司）
IBM Research AI, Almaden Research Center（IBM研究AI，阿尔马登研究中心）
In this working notes paper, we describe IBM Research AI (Almaden) team's participation in the ImageCLEF 2019 VQA-Med competition. The challenge consists of four question-answering tasks based on radiology images. The diversity of imaging modalities, organs and disease types combined with a small imbalanced training set made this a highly complex problem. To overcome these difficulties, we implemented a modular pipeline architecture that utilized transfer learning and multi-task learning. Our findings led to the development of a novel model called Supporting Facts Network (SFN). The main idea …
Contrastive Language--Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image--text pairs collected online. Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems. This work evaluates the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA). To this end, we present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles. Our experiments are conducted on two MedVQA benchmark datasets and investigate two MedVQA met…
MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
Faculty Of Information Technology VNU University of Engineering（信息科技学院越南工程大学）；IT-BT Convergence Technology Division Vietnam-Korea Institute of Science（IT-BT融合技术部门越南-韩国科学技术院）；TADI Global Lab TADI Global Company Limited（TADI全球实验室TADI全球有限公司）；Faculty of Finance Banking Academy of Vietnam（金融学院越南银行学院）
UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering
Zhejiang University（浙江大学）；University of Illinois Urbana–Champaign Institute（伊利诺伊大学厄巴纳-香槟分校研究院）；Sixth Medical Center of Chinese PLA General Hospital（中国人民解放军总医院第六医学中心）
Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
School of Computing；Artificial Intelligence Southwest Jiaotong University Chengdu, China；Artificial Intelligence Southwest Jiaotong University Chengdu, China z\ c\；Institute of Innovative Research Tokyo Institute of Technology Yokohama, Japan
Saliency Guided Longitudinal Medical Visual Question Answering
Dept. of Computer Science and Engineering University of California, San Diego（计算机科学与工程系，加州大学圣地亚哥分校）；Dept. of Radiology and Biomedical Imaging Yale University（放射学与生物医学成像系，耶鲁大学）
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
InViC: Intent-aware Visual Cues for Medical Visual Question Answering
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China（集成空天地海大数据应用技术国家工程实验室，计算机科学与工程学院，西北工业大学，西安710072，中国）；Westlake University（西湖大学）；Southern Medical University（南方医科大学）
Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering
Tandon School of Engineering, New York University（纽约大学工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）
National University of Singapore（新加坡国立大学）；Imperial College London（帝国理工学院）；National Institute of Technology, Tiruchirappalli（蒂鲁吉拉伯利国家理工学院）；Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Visual question answering (VQA) in surgery is largely unexplored. Expert surgeons are scarce and are often overloaded with clinical and academic workloads. This overload often limits their time answering questionnaires from patients, medical students or junior residents related to surgical procedures. At times, students and junior residents also refrain from asking too many questions during classes to reduce disruption. While computer-aided simulators and recording of past surgical procedures have been made available for them to observe and improve their skills, they still hugely rely on medic…
We introduce a novel privacy-preserving methodology for performing Visual Question Answering on the edge. Our method constructs a symbolic representation of the visual scene, using a low-complexity computer vision model that jointly predicts classes, attributes and predicates. This symbolic representation is non-differentiable, which means it cannot be used to recover the original image, thereby keeping the original image private. Our proposed hybrid solution uses a vision model which is more than 25 times smaller than the current state-of-the-art (SOTA) vision models, and 100 times smaller th…
Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
Johns Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia（意大利理工学院生物分子纳米技术中心）
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine（埃默里大学医学院放射肿瘤学系与温希普癌症研究所）；Department of Biostatistics and Bioinformatics, Emory University（埃默里大学生物统计学与生物信息学系）
KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Tianjin Central Hospital of Gynecology Obstetrics（天津市中心妇产科医院）；Department of Computer Science and Information Technology, La Trobe University（拉筹伯大学计算机科学与信息技术系）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Computer Software, Tianjin University（天津大学计算机软件学院）；Centre for Artificial Intelligence driven Drug Discovery, Faculty of Applied Science, Macao Polytechnic University（澳门理工大学应用科学学院人工智能驱动药物发现中心）；Department of Cardiology, Tianjin Chest Hospital（天津市胸科医院心内科）
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Mathematics and Computer Science, Faculty of Science, Alexandria University（亚历山大大学数学与计算机科学系）；Sheikh Shakhbout Medical City（谢赫·沙赫布OUT医疗城）
