Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model for the challenging problem of preoperative diagnosis of parotid gland tumors, and the experimental results show the advantages of our proposed method. We argue that the combination of CNN and transformer has tremendous potential in a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to medical image classification.
This paper presents a new approach of transfer learning-based medical image classification to mitigate insufficient labeled data problem in medical domain. Instead of direct transfer learning from source to small number of labeled target data, we propose a modality-bridge transfer learning which employs the bridge database in the same medical imaging acquisition modality as target database. By learning the projection function from source to bridge and from bridge to target, the domain difference between source (e.g., natural images) and target (e.g., X-ray images) can be mitigated. Experimenta…
Numerous factors could lead to partial deteriorations of medical images. For example, metallic implants will lead to localized perturbations in MRI scans. This will affect further post-processing tasks such as attenuation correction in PET/MRI or radiation therapy planning. In this work, we propose the inpainting of medical images via Generative Adversarial Networks (GANs). The proposed framework incorporates two patch-based discriminator networks with additional style and perceptual losses for the inpainting of missing information in realistically detailed and contextually consistent manner. …
M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model
School of Computer Science and Technique（计算机科学与技术学院）；Tongji University（同济大学）；School of Communications and Electronic Engineering（通讯与电子工程学院）；East China Normal University（华东师范大学）
KAIST（韩国科学技术院）；Asan Medical Center（峨山医学中心）；University of Ulsan College of Medicine（蔚山大学医学院）
Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performanc…
MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology（东方理工大学）；Shanghai Jiao Tong University（上海交通大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Khalifa University（哈利法大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；EPFL（洛桑联邦理工学院）
Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image
School of Computer Science and Engineering, Central South University, Changsha, 410083, China（计算机科学与工程学院，中南大学，长沙，410083，中国）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha, 410083, China（机器视觉与智能医学工程研究中心，中南大学，长沙，410083，中国）；Department of Radiation Oncology, Washington University in St. Louis, USA（放射肿瘤科，华盛顿大学圣路易斯分校，美国）；Department of Nuclear Medicine, Henri Becquerel Cancer Center, Rouen, France（核医学科，亨利·贝克勒尔癌症中心，鲁昂，法国）；University of Rouen-Normandy, AMIS - QuantIF UR 4108, F-76000, Rouen, France（鲁昂-诺曼底大学，AMIS - QuantIF UR 4108，法国，F-76000，鲁昂，法国）
University of Oxford（牛津大学）；Plexalis Ltd.（Plexalis有限公司）
This paper explores the use of self-supervised deep learning in medical imaging in cases where two scan modalities are available for the same subject. Specifically, we use a large publicly-available dataset of over 20,000 subjects from the UK Biobank with both whole body Dixon technique magnetic resonance (MR) scans and also dual-energy x-ray absorptiometry (DXA) scans. We make three contributions: (i) We introduce a multi-modal image-matching contrastive framework, that is able to learn to match different-modality scans of the same subject with high accuracy. (ii) Without any adaption, we sho…
The University of Texas Arlington（德克萨斯大学阿灵顿分校）；RIKEN（理化学研究所）；University of Tokyo（东京大学）；National Cancer Center Research Institute（国立癌症中心研究所）；National Library of Medicine - National Institutes of Health（美国国立卫生研究院国家医学图书馆）；Temple University（天普大学）
Medical visual question answering (VQA) aims to answer clinically relevant questions regarding input medical images. This technique has the potential to improve the efficiency of medical professionals while relieving the burden on the public health system, particularly in resource-poor countries. Existing medical VQA methods tend to encode medical images and learn the correspondence between visual features and questions without exploiting the spatial, semantic, or medical knowledge behind them. This is partially because of the small size of the current medical VQA dataset, which often includes…
Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration
DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Lab（湖畔实验室）；College of Computer Science and Technology, Zhejiang University, China（浙江大学计算机科学与技术学院）；Shengjing Hospital of China Medical University, China（中国医科大学附属盛京医院）
Medical imaging is a cornerstone of therapy and diagnosis in modern medicine. However, the choice of imaging modality for a particular theranostic task typically involves trade-offs between the feasibility of using a particular modality (e.g., short wait times, low cost, fast acquisition, reduced radiation/invasiveness) and the expected performance on a clinical task (e.g., diagnostic accuracy, efficacy of treatment planning and guidance). In this work, we aim to apply the knowledge learned from the less feasible but better-performing (superior) modality to guide the utilization of the more-fe…
A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning
School of Electrical and Control Engineering, North China University of Technology（北方工业大学电气与控制工程学院）；School of Automation and Electrical Engineering, Shenyang Ligong University（沈阳理工大学自动化与电气工程学院）；The Fourth Affiliated Hospital of China Medical University（中国医科大学附属第四医院）
ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images
The Hong Kong University of Sciences and Technology, Guangzhou（香港科学与技术大学（广州））；Beihang University（北航大学）；Shandong University（山东大学）；Hubei University（湖北大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；Australian National University（澳大利亚国立大学）
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
School of Physics, Peking University（北京大学物理学院）；School of Computer Science, Peking University（北京大学计算机学院）；Department of Radiotherapy, Peking University Cancer Hospital（北京大学肿瘤医院放疗科）；Department of Radiation Oncology, Peking University Third Hospital（北京大学第三医院放疗科）
École de technologie Superieure (ETS)（高等技术学院（ETS））；IIT Kharagpur（印度理工学院卡拉格布尔分校）
We address the problem of segmenting 3D multi-modal medical images in scenarios where very few labeled examples are available for training. Leveraging the recent success of adversarial learning for semi-supervised segmentation, we propose a novel method based on Generative Adversarial Networks (GANs) to train a segmentation model with both labeled and unlabeled images. The proposed method prevents over-fitting by learning to discriminate between true and fake patches obtained by a generator network. Our work extends current adversarial learning approaches, which focus on 2D single-modality ima…
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Fudan University, China（复旦大学）；University of Queensland, Australia（昆士兰大学）；Shanghai Academy of AI for Science, China（上海人工智能科学研究院）；Huashan Hospital, National Center for Neurological Disorders, Fudan University, China（华山医院，国家神经系统疾病中心，复旦大学）；Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore（生物信息研究所（BII），科技研究局（A*STAR），新加坡）
BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities
Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed大学人工智能学院）；Linköping University（林肯大学）；Shaikh Tahnoon bin Mohammed Medical City（Shaikh Tahnoon bin Mohammed医疗城）；Tawam Hospital（Tawam医院）；Sheikh Shakhbout Medical City（Sheikh Shakhbout医疗城）；Govt Medical College Kozhikode（科钦政府医学院）
No Modality Left Behind: Dynamic Model Generation for Incomplete Medical Data
Computational Imaging Research Lab（计算成像研究实验室）；Department for Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Medical University of Vienna（维也纳医学大学）；Comprehensive Center for Artificial Intelligence in Medicine（医学人工智能综合中心）；Christian Doppler Laboratory for Machine Learning Driven Precision Imaging（机器学习驱动精准成像的克里斯蒂安·多普勒实验室）；Department of Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉·A·马丁努斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Department of Radiology（放射科）；Division of General and Pediatric Radiology（普通和儿童放射科部门）
Prototype-Enhanced Confidence Modeling for Cross-Modal Medical Image-Report Retrieval
School of Computer Science, The University of Nottingham, NG8 1BB Nottingham, U.K.（计算机科学学院，诺丁汉大学）；Department of Intelligent Science, Xi’an Jiaotong-Liverpool University, China, 215123.（智能科学系，西安交通大学利物浦大学）
Embedding Radiomics into Vision Transformers for Multimodal Medical Image Classification
Duke Kunshan University（昆山杜克大学）；Jiangsu Provincial University Key (Construction) Laboratory for Smart Diagnosis and Treatment of Lung Cancer（江苏省高校重点（建设）实验室肺癌智能诊疗实验室）；The First People’s Hospital of Kunshan（昆山市第一人民医院）；Duke University（杜克大学）
Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network wei…
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
Being able to explain the prediction to clinical end-users is a necessity to leverage the power of AI models for clinical decision support. For medical images, saliency maps are the most common form of explanation. The maps highlight important features for AI model's prediction. Although many saliency map methods have been proposed, it is unknown how well they perform on explaining decisions on multi-modal medical images, where each modality/channel carries distinct clinical meanings of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for c…
While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, it is oftentimes very difficult to obtain a lot of labeled medical images for model training. In this paper, we study cross-modality data augmentation to mitigate the data deficiency issue in the medical imaging domain. We propose a discriminative unpaired image-to-image translation model which translates images in source modality into images in target modality where the translation task is condu…
ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Northwestern Polytechnical University（西北工业大学）
Multi-modal medical image completion has been extensively applied to alleviate the missing modality issue in a wealth of multi-modal diagnostic tasks. However, for most existing synthesis methods, their inferences of missing modalities can collapse into a deterministic mapping from the available ones, ignoring the uncertainties inherent in the cross-modal relationships. Here, we propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) in modeling and stochastically sampling a target probability distribution, and…
Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation
Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；AI 3 Fudan University（复旦大学）；Duke Kunshan University（杜克-昆山大学）；Zhongshan Hospital（中山医院）
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；City University of Hong Kong（香港城市大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；University of Georgia（佐治亚大学）；Massachusetts General Hospital（麻省总医院）
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA（系统工程系，史蒂文斯理工学院）；Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan（高雄长庚纪念医院核医学部）；Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan（国立高雄大学统计研究所）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（深圳先进技术研究院，中国科学院）
PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification
Dalian University of Technology（大连理工大学）；School of Computer Science and Technology（计算机科学与技术学院）；School of Computer Science and Engineering（计算机科学与工程学院）；Dalian Minzu University（大连民族大学）；School of Information and Communication Engineering（信息与通信工程学院）
MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis
Chair for AI for Image-Guided Diagnosis and Therapy, Technical University of Munich (TUM)（人工智能辅助影像诊断与治疗研究所，慕尼黑技术大学）；TUM University Hospital, Munich, Germany（慕尼黑技术大学医院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM)（人工智能在医疗与健康领域研究所，慕尼黑技术大学）；Department of Radiation Oncology, TUM University Hospital, Munich, Germany（放射肿瘤科，慕尼黑技术大学医院）；Chair for Computer Vision and Artificial Intelligence, Technical University of Munich (TUM)（计算机视觉与人工智能研究所，慕尼黑技术大学）；Department of Scientific Computing, Florida State University（科学计算系，佛罗里达州立大学）；Deutsches Konsortium für Translationale Krebsforschung (DKTK), Partner Site Munich（德国转化癌症研究联盟（DKTK）慕尼黑分部）；Institute of Radiation Medicine (IRM), Department of Radiation Sciences (DRS), Helmholtz Center Munich（放射医学研究所（IRM），辐射科学部门（DRS），海德堡中心慕尼黑）；Institute for Advanced Study, Technical University of Munich (TUM)（高级研究所，慕尼黑技术大学）
