AI 中文总结
研究在自回归电子健康记录基础模型中纳入多模态的方法，通过特定模态潜在压缩和门控交叉注意力框架，研究压缩单模态序列及预训练编码器选择对性能的影响，实验表明精心设计架构及临床评估的必要性。
AI 中文摘要
在经过标记化的电子健康记录（EHR）上训练的自回归基础模型可以支持零样本临床预测，但大多数模型仅对结构化事件代码进行操作，并未以有原则的方式纳入多种模态。我们提出了一个框架，使用特定模态的潜在压缩和带时间对齐的门控交叉注意力，在包括心电图波形、胸部X光图像和临床记录等辅助临床模态上对这类模型进行条件设定。我们研究了两个关键设计选择：一是在多模态交叉注意力之前如何压缩长的单模态序列；二是为每个模态选择预训练编码器如何影响下游性能。通过在MIMIC-IV上进行的受控消融实验，我们表明最佳的潜在压缩配置优于未压缩的交叉注意力和均值池化。编码器选择在模态内有明显影响，更强的预训练编码器始终优于较弱的替代方案。我们还表明，仅添加辅助模态并不能保证在ICU死亡率预测上优于仅使用EHR的基线。这意味着需要精心设计融合架构并在临床环境中进行适当评估。
英文摘要
Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.
Comments5 pages excluding references and supplements, 2 figures and 2 tables, Proceedings of the Workshop on Structured Data for Health at the 43rd International Conference on Machine Learning, Seoul, South Korea
Journal refICML2026 workshop on Structured Data for Health (SD4H)
