AI 中文总结
针对临床问题中多模态患者证据与外部知识缺乏显式关联的问题，提出MM-KG，通过显式对齐边链接患者数据与生物医学知识图谱，使临床大语言模型能检索、追溯并测试这些链接，显著提升需要双源知识的问答性能。
AI 中文摘要
临床问题通常依赖于将患者的多模态证据与外部生物医学知识联系起来，然而现有的预测系统很少显式地表示这种联系，因此既无法追溯到其证据来源，也无法移除这些联系以衡量其贡献。我们提出了MM-KG（多模态知识图谱），它将异质的、多模态的患者观察数据和生物医学概念表示为单一类型化图谱中的不同层，并通过显式的对齐边进行连接。首先，模态特定的协调器将EHR文本、影像、基因组和生物样本数据转换为映射到UMLS概念的典型化观察数据，然后由路由优先的对齐器将这些数据链接到生物医学知识图谱。随后，基于查询条件的检索为下游使用的大语言模型或图神经网络选择一个紧凑的子图。我们为MIMIC-IV和ADNI构建了MM-KG，并通过一个2x2设计来评估它们，该设计将患者证据、生物医学知识及其交互分开。在需要两种来源的问题上，单一来源的表现远未达到随机水平，而它们的组合在MIMIC上产生了+0.194的药物控制AUROC交互，在ADNI上产生了+0.299。在保留的五候选排序中，MM-KG在Hits@1上比MindMap高出+0.131，并在需要咨询患者的问题上优于改编的GraphCare，而从检索到的数据包中删除唯一包含答案的关系，Hits@1会回落到无知识基线的水平。最后，基于查询条件的检索以比最强通用策略少6.8倍的上下文达到了0.731的AUROC，而静态知识图谱上下文在普通结果预测上没有带来一致的增益。因此，知识图谱对临床大语言模型的益处不在于作为背景上下文，而在于作为多模态患者证据与问题所需关系之间的显式链接，而MM-KG使这些链接可检索、可追溯且可测试。
英文摘要
Clinical questions often depend on linking a patient's multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (Multimodal Knowledge Graph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and ADNI, and evaluate them with a 2x2 design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of +0.194 on MIMIC and +0.299 on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by +0.131 Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches 0.731 AUROC with 6.8x less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.
