MEDBind: Unifying Language and Multimodal Medical Data Embeddings
Peter Munk Cardiac Centre(彼得·芒克心脏中心);University Health Network (UHN)(大学网络健康中心);University of Toronto (U of T)(多伦多大学);Ted Rogers Centre for Heart Research(泰德·罗杰斯心脏研究中心);Vector Institute(向量研究所);Toronto General Hospital Research Institute(多伦多总医院研究所);Toronto General Hospital(多伦多总医院);Department of Medical Imaging(医学影像系)
Yuan Gao, Sangwook Kim, David E Austin, Chris McIntosh
Medical vision-language pretraining models (VLPM) have achieved remarkable progress in fusing chest X-rays (CXR) with clinical texts, introducing image-text data binding approaches that enable zero-shot learning and downstream clinical tasks. However, the current landscape lacks the holistic integration of additional medical modalities, such as electrocardiograms (ECG). We present MEDBind (Medical Electronic patient recorD), which learns joint embeddings across CXR, ECG, and medical text. Using text data as the central anchor, MEDBind features tri-modality binding, delivering competitive performance in top-K retrieval, zero-shot, and few-shot benchmarks against established VLPM, and the ability for CXR-to-ECG zero-shot classification and retrieval. This seamless integration is achieved through combination of contrastive loss on modality-text pairs with our proposed contrastive loss function, Edge-Modality Contrastive Loss, fostering a cohesive embedding space for CXR, ECG, and text. Finally, we demonstrate that MEDBind can improve downstream tasks by directly integrating CXR and ECG embeddings into a large-language model for multimodal prompt tuning.
Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture
Sichuan University(四川大学)
自动数据增强是近年来研究的热点,其目的是自动制定工程增强策略。许多以前的自动增强方法利用密度匹配策略,通过评估测试时增强性能方面的政策。在本文中,我们从理论上和经验证明之间的不一致性的训练和验证集的小规模医学图像数据集,称为域内采样偏差。接下来,我们证明了域内采样偏差可能导致密度匹配的效率低下。针对这一问题,提出了一种改进的增广搜索策略--增广密度匹配(Augmented Density Matching),该策略从先验分布中随机抽取策略进行训练。此外,通过统一数据增强和神经结构的搜索,提出了一种高效的自动机器学习(AutoML)算法。实验结果表明,所提出的方法优于最先进的方法MedMNIST,一个开创性的基准设计的AutoML在医学图像分析。
UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis
Shanghai Artificial Intelligence Laboratory;Shanghai Innovation Institute;Shanghai Jiao Tong University;Shanghai Institute of Optics;Fudan University;University of Cambridge;Monash University;DAMO Academy, Alibaba Group;Imperial College London;The University of Hong Kong;The Hong Kong University of Science;Hupan Lab;The Chinese University of Hong Kong
UniH$^3$: Unifying Hierarchical Homogeneity and Heterogeneity for All-in-One Medical Image Restoration
School of Biological Science and Medical Engineering, Beihang University(北京航空航天大学生物科学与医学工程学院);Department of Biomedical Engineering, Tsinghua University(清华大学生物医学工程系)
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);The Hong Kong University of Science and Technology(香港科技大学);Sun Yat-sen University(中山大学)
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Alibaba(阿里巴巴);The Hong Kong University of Science and Technology(香港科技大学);National University of Singapore(新加坡国立大学)
MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariant
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZIU-UIUC Institute, Zhejiang University(浙江大学ZIU-UIUC研究院);School of Public Health, Zhejiang University(浙江大学公共卫生学院)
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University(俄亥俄州立大学);Imperial College London(伦敦帝国学院);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Peking University(北京大学)
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
MedVisAI Lab(MedVisAI实验室);ECE, Northwestern University(电子工程系,西北大学);Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究所(I2R),A*STAR,新加坡);Lee Kong Chian School of Medicine, Nanyang Technological University(Lee Kong Chian医学院,南洋理工大学)
Unifying and Personalizing Weakly-supervised Federated Medical Image Segmentation via Adaptive Representation and Aggregation
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Jiaxing Research Institute, Southern University of Science and Technology(南方科技大学嘉兴研究院)
MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
Technical University of Munich (TUM)(慕尼黑工业大学(TUM));TUM University Hospital(慕尼黑工业大学医院);University of Oxford(牛津大学);Imperial College London(伦敦帝国学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Sheffield(谢菲尔德大学)
One Framework to Rule Them All: Unifying Multimodal Tasks with LLM Neural-Tuning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);College of Information Science and Engineering, Ritsumeikan University(立命馆大学信息科学与工程学院)
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
Department of Biomedical Engineering, University of Florida(佛罗里达大学生物医学工程系);Department of Radiology, University of Florida(佛罗里达大学放射学系);Research Computing, University of Florida(佛罗里达大学研究计算中心);Department of Medicine, University of Florida(佛罗里达大学医学系);Department of Radiology, UC San Francisco(旧金山大学放射学系)
Learning Large Margin Sparse Embeddings for Open Set Medical Diagnosis
Beihang University(北京航空航天大学);School of Biological Science and Medical Engineering(生物与医学工程学院);Hefei Innovation Research Institute, Beihang University(北京航空航天大学合肥创新研究院)
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine(放射肿瘤科,Winship癌症研究所,埃默里大学医学院);Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology(生物医学工程系,工程学院,佐治亚理工学院);Department of Computer Science and Mathematics, Laney Graduate School, Emory University(计算机科学与数学系,兰尼研究生院,埃默里大学);School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology(电气与计算机工程系,工程学院,佐治亚理工学院)