Understanding 3D medical image volumes is a critical task in the medical domain. However, existing 3D convolution and transformer-based methods have limited semantic understanding of an image volume and also need a large set of volumes for training. Recent advances in multi-modal large language models (MLLMs) provide a new and promising way to understand images with the help of text descriptions. However, most current MLLMs are designed for 2D natural images. To enhance the 3D medical image understanding with 2D MLLMs, we propose a novel pre-training framework called Med3DInsight, which marries existing 3D image encoders with 2D MLLMs and bridges them via a designed Plane-Slice-Aware Transformer (PSAT) module. Extensive experiments demonstrate our SOTA performance on two downstream segmentation and classification tasks, including three public datasets with CT and MRI modalities and comparison to more than ten baselines. Med3DInsight can be easily integrated into any current 3D medical image understanding network and improves its performance by a good margin.
This paper presents a new approach of transfer learning-based medical image classification to mitigate insufficient labeled data problem in medical domain. Instead of direct transfer learning from source to small number of labeled target data, we propose a modality-bridge transfer learning which employs the bridge database in the same medical imaging acquisition modality as target database. By learning the projection function from source to bridge and from bridge to target, the domain difference between source (e.g., natural images) and target (e.g., X-ray images) can be mitigated. Experimenta…
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependenci…
Numerous factors could lead to partial deteriorations of medical images. For example, metallic implants will lead to localized perturbations in MRI scans. This will affect further post-processing tasks such as attenuation correction in PET/MRI or radiation therapy planning. In this work, we propose the inpainting of medical images via Generative Adversarial Networks (GANs). The proposed framework incorporates two patch-based discriminator networks with additional style and perceptual losses for the inpainting of missing information in realistically detailed and contextually consistent manner. …
M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model
School of Computer Science and Technique（计算机科学与技术学院）；Tongji University（同济大学）；School of Communications and Electronic Engineering（通讯与电子工程学院）；East China Normal University（华东师范大学）
KAIST（韩国科学技术院）；Asan Medical Center（峨山医学中心）；University of Ulsan College of Medicine（蔚山大学医学院）
Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performanc…
MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology（东方理工大学）；Shanghai Jiao Tong University（上海交通大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Khalifa University（哈利法大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；EPFL（洛桑联邦理工学院）
Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image
School of Computer Science and Engineering, Central South University, Changsha, 410083, China（计算机科学与工程学院，中南大学，长沙，410083，中国）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha, 410083, China（机器视觉与智能医学工程研究中心，中南大学，长沙，410083，中国）；Department of Radiation Oncology, Washington University in St. Louis, USA（放射肿瘤科，华盛顿大学圣路易斯分校，美国）；Department of Nuclear Medicine, Henri Becquerel Cancer Center, Rouen, France（核医学科，亨利·贝克勒尔癌症中心，鲁昂，法国）；University of Rouen-Normandy, AMIS - QuantIF UR 4108, F-76000, Rouen, France（鲁昂-诺曼底大学，AMIS - QuantIF UR 4108，法国，F-76000，鲁昂，法国）
University of Oxford（牛津大学）；Plexalis Ltd.（Plexalis有限公司）
This paper explores the use of self-supervised deep learning in medical imaging in cases where two scan modalities are available for the same subject. Specifically, we use a large publicly-available dataset of over 20,000 subjects from the UK Biobank with both whole body Dixon technique magnetic resonance (MR) scans and also dual-energy x-ray absorptiometry (DXA) scans. We make three contributions: (i) We introduce a multi-modal image-matching contrastive framework, that is able to learn to match different-modality scans of the same subject with high accuracy. (ii) Without any adaption, we sho…
The University of Texas Arlington（德克萨斯大学阿灵顿分校）；RIKEN（理化学研究所）；University of Tokyo（东京大学）；National Cancer Center Research Institute（国立癌症中心研究所）；National Library of Medicine - National Institutes of Health（美国国立卫生研究院国家医学图书馆）；Temple University（天普大学）
Medical visual question answering (VQA) aims to answer clinically relevant questions regarding input medical images. This technique has the potential to improve the efficiency of medical professionals while relieving the burden on the public health system, particularly in resource-poor countries. Existing medical VQA methods tend to encode medical images and learn the correspondence between visual features and questions without exploiting the spatial, semantic, or medical knowledge behind them. This is partially because of the small size of the current medical VQA dataset, which often includes…
Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration
DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Lab（湖畔实验室）；College of Computer Science and Technology, Zhejiang University, China（浙江大学计算机科学与技术学院）；Shengjing Hospital of China Medical University, China（中国医科大学附属盛京医院）
Medical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice of imaging modality for a particular theranostic task typically involves trade-offs between the feasibility of using a particular modality (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) and the expected performance on a clinical task (e.g., diagnostic accuracy, efficacy of treatment planning and guidance). In this work, we aim to apply the knowledge learned from the less feasible but better-performing (superior) modality to guide the utilization of the more-fe…
A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning
School of Electrical and Control Engineering, North China University of Technology（北方工业大学电气与控制工程学院）；School of Automation and Electrical Engineering, Shenyang Ligong University（沈阳理工大学自动化与电气工程学院）；The Fourth Affiliated Hospital of China Medical University（中国医科大学附属第四医院）
ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images
The Hong Kong University of Sciences and Technology, Guangzhou（香港科学与技术大学（广州））；Beihang University（北航大学）；Shandong University（山东大学）；Hubei University（湖北大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；Australian National University（澳大利亚国立大学）
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
School of Physics, Peking University（北京大学物理学院）；School of Computer Science, Peking University（北京大学计算机学院）；Department of Radiotherapy, Peking University Cancer Hospital（北京大学肿瘤医院放疗科）；Department of Radiation Oncology, Peking University Third Hospital（北京大学第三医院放疗科）
The visual analysis of retina and of its vascular characteristics is important in the diagnosis and monitoring of diseases of visual perception. In the related medical diagnoses, the digital processing of the fundus images is used to obtain the segmentation of retinal vessels. However, an image segmentation is often requiring methods based on peculiar or complex algorithms: in this paper we will show some alternative approaches obtained by applying freely available tools to enhance, without a specific segmentation, the images of the fundus of the eye. We will see in particular, that combining …
École de technologie Superieure (ETS)（高等技术学院（ETS））；IIT Kharagpur（印度理工学院卡拉格布尔分校）
We address the problem of segmenting 3D multi-modal medical images in scenarios where very few labeled examples are available for training. Leveraging the recent success of adversarial learning for semi-supervised segmentation, we propose a novel method based on Generative Adversarial Networks (GANs) to train a segmentation model with both labeled and unlabeled images. The proposed method prevents over-fitting by learning to discriminate between true and fake patches obtained by a generator network. Our work extends current adversarial learning approaches, which focus on 2D single-modality ima…
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Fudan University, China（复旦大学）；University of Queensland, Australia（昆士兰大学）；Shanghai Academy of AI for Science, China（上海人工智能科学研究院）；Huashan Hospital, National Center for Neurological Disorders, Fudan University, China（华山医院，国家神经系统疾病中心，复旦大学）；Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore（生物信息研究所（BII），科技研究局（A*STAR），新加坡）
BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities
Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed大学人工智能学院）；Linköping University（林肯大学）；Shaikh Tahnoon bin Mohammed Medical City（Shaikh Tahnoon bin Mohammed医疗城）；Tawam Hospital（Tawam医院）；Sheikh Shakhbout Medical City（Sheikh Shakhbout医疗城）；Govt Medical College Kozhikode（科钦政府医学院）
While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, it is oftentimes very difficult to obtain a lot of labeled medical images for model training. In this paper, we study cross-modality data augmentation to mitigate the data deficiency issue in the medical imaging domain. We propose a discriminative unpaired image-to-image translation model which translates images in source modality into images in target modality where the translation task is condu…
Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network wei…
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
Being able to explain the prediction to clinical end-users is a necessity to leverage the power of AI models for clinical decision support. For medical images, saliency maps are the most common form of explanation. The maps highlight important features for AI model's prediction. Although many saliency map methods have been proposed, it is unknown how well they perform on explaining decisions on multi-modal medical images, where each modality/channel carries distinct clinical meanings of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for c…
Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data
Columbia University（哥伦比亚大学）；Texas Tech University（德克萨斯理工大学）；University of California, Davis（加利福尼亚大学戴维斯分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；North Carolina State University（北卡罗来纳州立大学）
University of Amsterdam（阿姆斯特丹大学）；Georgia Institute of Technology（佐治亚理工学院）
Medical image captioning automatically generates a medical description to describe the content of a given medical image. A traditional medical image captioning model creates a medical description only based on a single medical image input. Hence, an abstract medical description or concept is hard to be generated based on the traditional approach. Such a method limits the effectiveness of medical image captioning. Multi-modal medical image captioning is one of the approaches utilized to address this problem. In multi-modal medical image captioning, textual input, e.g., expert-defined keywords, …
ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Northwestern Polytechnical University（西北工业大学）
Multi-modal medical image completion has been extensively applied to alleviate the missing modality issue in a wealth of multi-modal diagnostic tasks. However, for most existing synthesis methods, their inferences of missing modalities can collapse into a deterministic mapping from the available ones, ignoring the uncertainties inherent in the cross-modal relationships. Here, we propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) in modeling and stochastically sampling a target probability distribution, and…
Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation
Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；AI 3 Fudan University（复旦大学）；Duke Kunshan University（杜克-昆山大学）；Zhongshan Hospital（中山医院）
Rutgers University（罗格斯大学）；SenseBrain Research（SenseBrain研究院）；National Heart Centre Singapore（新加坡国家心脏中心）；Duke-National University Of Singapore（杜克-新加坡国立大学）
Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose a privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could outpu…
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；City University of Hong Kong（香港城市大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；University of Georgia（佐治亚大学）；Massachusetts General Hospital（麻省总医院）
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA（系统工程系，史蒂文斯理工学院）；Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan（高雄长庚纪念医院核医学部）；Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan（国立高雄大学统计研究所）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（深圳先进技术研究院，中国科学院）
PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification
Dalian University of Technology（大连理工大学）；School of Computer Science and Technology（计算机科学与技术学院）；School of Computer Science and Engineering（计算机科学与工程学院）；Dalian Minzu University（大连民族大学）；School of Information and Communication Engineering（信息与通信工程学院）
MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis
Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)（人工智能辅助影像诊断与治疗研究所，慕尼黑技术大学）；TUM University Hospital, Munich, Germany（慕尼黑技术大学医院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)（人工智能在医疗与健康领域研究所，慕尼黑技术大学）；Department of Radiation Oncology, TUM University Hospital, Munich, Germany（放射肿瘤科，慕尼黑技术大学医院）；Chair for Computer Vision and Artificial Intelligence, Technical University of Munich (TUM)（计算机视觉与人工智能研究所，慕尼黑技术大学）；Department of Scientific Computing, Florida State University（科学计算系，佛罗里达州立大学）；Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich（德国转化癌症研究联盟（DKTK）慕尼黑分部）；Institute of Radiation Medicine (IRM), Department of Radiation Sciences (DRS), Helmholtz Center Munich（放射医学研究所（IRM），辐射科学部门（DRS），海德堡中心慕尼黑）；Institute for Advanced Study, Technical University of Munich (TUM)（高级研究所，慕尼黑技术大学）
Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；Sir Run Run Shaw Hospital（邵逸夫医院）；Lenovo Research（联想研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang Normal University（浙江师范大学）
Deep learning models, such as the fully convolutional network (FCN), have been widely used in 3D biomedical segmentation and achieved state-of-the-art performance. Multiple modalities are often used for disease diagnosis and quantification. Two approaches are widely used in the literature to fuse multiple modalities in the segmentation networks: early-fusion (which stacks multiple modalities as different input channels) and late-fusion (which fuses the segmentation results from different modalities at the very end). These fusion methods easily suffer from the cross-modal interference caused by…
