arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07016cs.AI
收藏

REALM:基于RAG驱动的大语言模型多模态电子健康记录分析增强

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Software, Beihang University(北京航空航天大学软件学院);China Mobile Research Institute(中国移动研究院)

Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan

AI总结:

提出REALM框架,利用RAG技术结合大语言模型和GRU编码多模态EHR数据,通过匹配外部知识图谱PrimeKG提取医疗知识并自适应融合,在MIMIC-III任务上有效提升了临床预测性能。

摘要:

多模态电子健康记录(EHR)数据的整合显著提升了临床预测能力。利用临床笔记和多元时间序列EHR,现有模型通常缺乏与临床任务相关的医疗上下文,促使人们引入外部知识,特别是来自知识图谱(KG)的知识。以往使用KG知识的方法主要侧重于结构化知识提取,忽略了非结构化数据模态和语义高维医疗知识。为此,我们提出了REALM,一个由检索增强生成(RAG)驱动的框架,用于增强多模态EHR表示以解决这些局限性。首先,我们应用大语言模型(LLM)编码长上下文临床笔记,并使用GRU模型编码时间序列EHR数据。其次,我们提示LLM提取与任务相关的医疗实体,并在专业标注的外部知识图谱(PrimeKG)中匹配实体及对应的医疗知识。通过匹配并与临床标准对齐,我们的框架消除了幻觉并确保了一致性。最后,我们提出了一种自适应多模态融合网络,将提取的知识与多模态EHR数据集成。我们在MIMIC-III死亡率和再入院任务上的大量实验表明,REALM框架优于基线,强调了每个模块的有效性。REALM框架有助于改善多模态EHR数据在医疗保健中的应用,并弥合知情临床预测所需细致医疗上下文的差距。

英文摘要:

The integration of multimodal Electronic Health Records (EHR) data has significantly improved clinical predictive capabilities. Leveraging clinical notes and multivariate time-series EHR, existing models often lack the medical context relevent to clinical tasks, prompting the incorporation of external knowledge, particularly from the knowledge graph (KG). Previous approaches with KG knowledge have primarily focused on structured knowledge extraction, neglecting unstructured data modalities and semantic high dimensional medical knowledge. In response, we propose REALM, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR representations that address these limitations. Firstly, we apply Large Language Model (LLM) to encode long context clinical notes and GRU model to encode time-series EHR data. Secondly, we prompt LLM to extract task-relevant medical entities and match entities in professionally labeled external knowledge graph (PrimeKG) with corresponding medical knowledge. By matching and aligning with clinical standards, our framework eliminates hallucinations and ensures consistency. Lastly, we propose an adaptive multimodal fusion network to integrate extracted knowledge with multimodal EHR data. Our extensive experiments on MIMIC-III mortality and readmission tasks showcase the superior performance of our REALM framework over baselines, emphasizing the effectiveness of each module. REALM framework contributes to refining the use of multimodal EHR data in healthcare and bridging the gap with nuanced medical context essential for informed clinical predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑