M2G-LLM：通过多模态图推理和LLM上下文注入增强临床预测

中文摘要 AI 辅助

整合多样化的数据模态——如临床笔记、实验室结果和医学影像——对于推进临床决策至关重要。尽管大型语言模型（LLMs）在处理非结构化临床文本方面表现出色，但其在纳入非文本模态方面的能力有限，阻碍了其在医疗保健应用中的更广泛实用性。在此，我们介绍了M2G-LLM（多模态医学图-LLM），一种新颖的框架，通过图神经网络（GNNs）增强LLMs的多模态集成与对齐。我们的方法对患者就诊之间的时间关系进行建模，在临床相似患者之间传播信息，并对齐异构数据源以构建丰富的多模态上下文向量。这些向量被注入到LLM的中间层，实现对文本和非文本模态的联合推理。我们在MIMIC-IV和MIMIC-CXR数据集上评估了M2G-LLM，展示了在临床预测任务上相较于强基线模型的改进。我们的结果凸显了将LLMs的语言理解能力与GNNs的关系推理能力相结合，用于全面的多模态医疗分析的潜力。

英文摘要

Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.

发表机构

• University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
• University of Pennsylvania（宾夕法尼亚大学）

机构由 AI 辅助整理，请以论文原文为准。