Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review aims to comprehensively analyze and formalize current intermediate fusion methods in biomedical applications. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.
MMCTOP: A Multimodal Textualization and Mixture-of-Experts Framework for Clinical Trial Outcome Prediction
Nova School of Business and Economics(诺瓦商学院);Hogarthian Technologies(霍加斯技术);School of Medicine(医学院);Oregon Health & Science University(俄勒冈健康与科学大学);Cleveland Clinic(克利夫兰诊所);IBM Research(IBM研究院)
Multi-task Joint Strategies of Self-supervised Representation Learning on Biomedical Networks for Drug Discovery
College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机科学与电子工程学院);Institute of Open Source Software and Platform, Peng Cheng Laboratory(鹏城实验室开源软件与平台研究所);Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);The State Key Laboratory of Chemo/Biosensing and Chemometrics, Hunan University(湖南大学化学生物传感与计量学国家重点实验室)
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm(法国国家健康与医学研究院);University of Western Brittany(西布列塔尼大学);IMT Atlantique(大西洋电信学院);Sorbonne University(索邦大学);Lariboisière Hospital(拉里布瓦西耶尔医院);AP-HP(巴黎公共医疗集团);Paris Cité University(巴黎城市大学);CHRU Brest(布雷斯特大学中心医院);LaTIM UMR 1101(LaTIM联合研究单位1101)
Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education East China University of Science and Technology(能源化工过程智能制造重点实验室,东华大学);Research Institute of Intelligent Control and Systems Harbin Institute of Technology(智能控制与系统研究室,哈尔滨工业大学);Department of Emergency Medicine, Sir Run Run Shaw Hospital Zhejiang University School of Medicine(浙江大学医学院急诊医学科);Provincial Key Laboratory of Precise Diagnosis Treatment of Abdominal Infection, Sir Run Run Shaw Hospital Zhejiang University School of Medicine(腹部感染精准诊断治疗省级重点实验室,浙江大学医学院);School of Medicine Shaoxing University(绍兴大学医学院)
Doctor Sun是一种双语多模态大语言模型,通过整合预训练视觉编码器和医学LLM,提升生物医学多模态任务的性能,并提供SunMed-VL数据集支持研究进展。
CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs
Technology, Innovation, Entrepreneurship Department(技术、创新与创业部门);King Abdullah University of Science and Technology(卡塔尔科技大学);School of Computing and Information Systems(计算与信息系统学院);The University of Melbourne(墨尔本大学);College of Computer Science and Technology(计算机科学与技术学院);Jilin University(吉林大学);School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学)
Causal Representation Learning from Multimodal Biomedical Observations
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);University of Bristol(布里斯托大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所)
Multimodal Fusion on Low-quality Data: A Comprehensive Survey
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);College of Computer Science, Sichuan University(四川大学计算机学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院)
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Department of Computer Science(计算机科学系);Hong Kong Baptist University(香港 Baptist 大学);Division of Artificial Intelligence(人工智能 division);Lingnan University(Lingnan 大学);School of Nursing(护理学院);The Hong Kong Polytechnic University(香港理工大学)
Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning
School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学);School of Automation, Central South University(自动化学院,中南大学);School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学)
MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions
VNU University of Engineering and Technology (VNU-UET)(越南工程技术大学);Lausanne University Hospital (CHUV)(洛桑大学医院);University of Lausanne (UNIL)(洛桑大学);Hanoi Medical University(河内医学院);National Geriatric Hospital, Hanoi(河内国立老年医院)
Variational Fusion for Multimodal Sentiment Analysis
A*STAR AI Initiative, Institute of High Performance Computing(A*STAR人工智能计划、高性能计算研究院);Adobe Research(奥多比研究院);University of Michigan(密歇根大学);Centro de Investigación en Computación, Instituto Politécnico Nacional(国立理工学院计算研究中心);SUTD(新加坡科技设计大学)
ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
Southwest Jiaotong University(西南交通大学);Microsoft Research Asia(微软亚洲研究院);Southwestern University of Finance and Economics(西南财经大学);Ocean University of China(中国海洋大学);Tencent(腾讯)
Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data
College of Science and Engineering, Hamad Bin Khalifa University(哈马德·本·哈利法大学科学与工程学院);College of Health and Life Sciences, Hamad Bin Khalifa University(哈马德·本·哈利法大学健康与生命科学学院)
From Tissue Plane to Organ World: A Benchmark Dataset for Multimodal Biomedical Image Registration using Deep Co-Attention Networks
Carnegie Mellon University(卡内基梅隆大学);The Hong Kong Polytechnic University(香港理工大学);University of Pittsburgh(匹兹堡大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Anhui University(安徽大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);University of Science and Technology of China(中国科学技术大学);Anhui Medical University(安徽医科大学)