GEM：利用时间序列与图像赋能 MLLM 实现基于证据定位的 ECG 理解
GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images
AI总结：
本文提出 GEM，融合 ECG 时间序列、12 导联图像与文本，并通过双编码器、跨模态对齐和 ECG-Grounding 数据提升诊断、可解释性与证据定位能力。
摘要：
尽管近期的多模态大语言模型（MLLM）推动了自动化 ECG 解读，但它们仍面临两个关键局限：（1）时间序列信号与可视化 ECG 表征之间的多模态协同不足；（2）在将诊断与细粒度波形证据关联方面的可解释性有限。我们提出 GEM，这是首个统一 ECG 时间序列、12 导联 ECG 图像和文本的 MLLM，用于实现基于证据定位且符合临床医生习惯的 ECG 解读。GEM 通过三项核心创新实现特征定位分析、证据驱动推理和类临床医生诊断流程：提取互补时间序列与图像特征的双编码器框架、用于有效多模态理解的跨模态对齐，以及用于生成高粒度定位数据（ECG-Grounding）的知识引导指令生成，该数据将诊断与可测量参数（例如 QRS/PR 间期）关联。此外，我们提出 Grounded ECG Understanding 任务，这是一个具有临床动机的基准，旨在全面评估 MLLM 的基于证据定位 ECG 理解能力。在现有基准和我们提出的基准上的实验结果表明，GEM 显著提升了预测性能（CSN 提升 7.4%）、可解释性（提升 22.7%）和证据定位能力（提升 24.8%），使其更适合真实世界临床应用。GitHub 仓库：https://github.com/lanxiang1017/GEM.git
英文摘要：
While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between time series signals and visual ECG representations, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters ($e.g.$, QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN $7.4\% \uparrow$), explainability ($22.7\% \uparrow$), and grounding ($24.8\% \uparrow$), making it more suitable for real-world clinical applications. GitHub repository: https://github.com/lanxiang1017/GEM.git
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
