arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21950cs.CL
收藏

MEDSYN:多证据合成在复杂临床病例中的基准测试用于多模态大语言模型

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

ETH Zurich(苏黎世联邦理工学院);MBZUAI(马克斯·普朗克人工智能研究所);Amazon(亚马逊);University of Oxford(牛津大学);University of Bern(伯尔尼大学);Imperial College London(伦敦帝国理工学院);Peking University(北京大学)

Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Bang Zheng, Kyle Lam, Lin Li, Jianing Qiu

AI总结:

MEDSYN是一个多语言、多模态的复杂临床病例基准测试,用于评估多模态大语言模型在差分诊断和最终诊断中的表现,揭示模型在异质临床证据合成中的不足。

摘要:

多模态大语言模型(MLLMs)在医疗应用中展现出巨大潜力,但现有基准测试无法充分捕捉真实世界临床复杂性。我们介绍了MEDSYN,一个包含多达7种不同视觉临床证据(CE)类型的多语言、多模态基准测试。模仿临床工作流程,我们评估了18个MLLMs在差分诊断(DDx)生成和最终诊断(FDx)选择上的表现。尽管顶级模型在DDx生成上往往匹配甚至超越人类专家,但所有MLLMs在DDx-FDx性能差距上均远大于专家医生,表明在异质CE类型合成中存在失败模式。消融分析将此失败归因于(i)过度依赖判别力较弱的文本CE(如医疗史)以及(ii)跨模态CE利用差距。我们引入了证据敏感度来量化后者,并显示较小的差距与更高的诊断准确性相关。最后,我们展示了如何利用它来指导改进模型性能。我们将开源我们的基准和代码。

英文摘要:

Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity. We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case. Mirroring clinical workflow, we evaluate 18 MLLMs on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. While top models often match or even outperform human experts on DDx generation, all MLLMs exhibit a much larger DDx--FDx performance gap compared to expert clinicians, indicating a failure mode in synthesis of heterogeneous CE types. Ablations attribute this failure to (i) overreliance on less discriminative textual CE ($\it{e.g.}$, medical history) and (ii) a cross-modal CE utilization gap. We introduce Evidence Sensitivity to quantify the latter and show that a smaller gap correlates with higher diagnostic accuracy. Finally, we demonstrate how it can be used to guide interventions to improve model performance. We will open-source our benchmark and code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑