arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.12894cs.CV
收藏

MEDBind:统一语言与多模态医疗数据嵌入

MEDBind: Unifying Language and Multimodal Medical Data Embeddings

Peter Munk Cardiac Centre(彼得·芒克心脏中心);University Health Network (UHN)(大学网络健康中心);University of Toronto (U of T)(多伦多大学);Ted Rogers Centre for Heart Research(泰德·罗杰斯心脏研究中心);Vector Institute(向量研究所);Toronto General Hospital Research Institute(多伦多总医院研究所);Toronto General Hospital(多伦多总医院);Department of Medical Imaging(医学影像系)

Yuan Gao, Sangwook Kim, David E Austin, Chris McIntosh

AI总结:

提出MEDBind模型,以文本为中心锚点结合对比损失实现CXR、ECG与文本的三模态绑定,在零样本和少样本任务中表现优异,并能通过多模态提示微调提升下游任务性能。

摘要:

医疗视觉语言预训练模型(VLPM)在融合胸部X射线(CXR)与临床文本方面取得了显著进展,引入了图像-文本数据绑定方法,实现了零样本学习和下游临床任务。然而,当前领域缺乏对心电图(ECG)等额外医疗模态的整体整合。我们提出了MEDBind(Medical Electronic patient recorD),学习CXR、ECG和医疗文本间的联合嵌入。以文本数据为中心锚点,MEDBind具有三模态绑定特性,在top-K检索、零样本和少样本基准测试中,与现有VLPM相比表现出极具竞争力的性能,并具备CXR到ECG的零样本分类与检索能力。这种无缝整合是通过将模态-文本对的对比损失与我们提出的对比损失函数Edge-Modality Contrastive Loss相结合来实现的,为CXR、ECG和文本构建了具有凝聚力的嵌入空间。最后,我们证明MEDBind能通过将CXR和ECG嵌入直接整合到大型语言模型中进行多模态提示微调,从而改进下游任务。

英文摘要:

Medical vision-language pretraining models (VLPM) have achieved remarkable progress in fusing chest X-rays (CXR) with clinical texts, introducing image-text data binding approaches that enable zero-shot learning and downstream clinical tasks. However, the current landscape lacks the holistic integration of additional medical modalities, such as electrocardiograms (ECG). We present MEDBind (Medical Electronic patient recorD), which learns joint embeddings across CXR, ECG, and medical text. Using text data as the central anchor, MEDBind features tri-modality binding, delivering competitive performance in top-K retrieval, zero-shot, and few-shot benchmarks against established VLPM, and the ability for CXR-to-ECG zero-shot classification and retrieval. This seamless integration is achieved through combination of contrastive loss on modality-text pairs with our proposed contrastive loss function, Edge-Modality Contrastive Loss, fostering a cohesive embedding space for CXR, ECG, and text. Finally, we demonstrate that MEDBind can improve downstream tasks by directly integrating CXR and ECG embeddings into a large-language model for multimodal prompt tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑