arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11217cs.CLcs.CV
收藏

医学多模态大语言模型的多维度评估基准

A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models

The City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen University(深圳大学);National Yang Ming Chiao Tung University(国立阳明交通大学);Taipei Veterans General Hospital(台北荣民总医院);Stanford University(斯坦福大学)

Jie Liu, Wenxuan Wang, Yihang Su, Jingyuan Huan, Wenting Chen, Yudi Zhang, Cheng-Yi Li, Kao-Jung Chang, Xiaohan Xin, Linlin Shen, Michael R. Lyu

AI总结:

针对现有基准不适用医学多模态大语言模型(Med-MLLMs)的问题,提出Asclepius基准,从多专科和多诊断能力评估模型,分析6个模型并与3名人类专家比较,为临床部署提供支撑。

摘要:

医学多模态大语言模型(Med-MLLMs)的重大突破通过强大的信息合成和医疗决策支持革新了现代医疗。然而,由于跨不同专科的真实世界诊断的复杂性,这些模型常被评估于不适合Med-MLLMs的基准上。为填补这一空白,我们引入Asclepius——一个新型Med-MLLM基准,从不同医学专科(心血管、胃肠病学等)和不同诊断能力(感知、疾病分析等)方面全面评估Med-MLLMs。基于3项核心原则,Asclepius通过涵盖15个医学专科、分层为3个主要类别和8个子类别临床任务、避免与现有VQA数据集重叠,确保全面评估。我们进一步深入分析6个Med-MLLMs并与3名人类专家比较,提供其在不同医疗场景下能力和局限性的见解。我们的工作不仅推进对Med-MLLMs能力的理解,还为未来评估及这些模型在临床环境的安全部署树立先例。

英文摘要:

The significant breakthroughs of Medical Multi-Modal Large Language Models (Med-MLLMs) renovate modern healthcare with robust information synthesis and medical decision support. However, these models are often evaluated on benchmarks that are unsuitable for the Med-MLLMs due to the complexity of real-world diagnostics across diverse specialties. To address this gap, we introduce Asclepius, a novel Med-MLLM benchmark that comprehensively assesses Med-MLLMs in terms of: distinct medical specialties (cardiovascular, gastroenterology, etc.) and different diagnostic capacities (perception, disease analysis, etc.). Grounded in 3 proposed core principles, Asclepius ensures a comprehensive evaluation by encompassing 15 medical specialties, stratifying into 3 main categories and 8 sub-categories of clinical tasks, and exempting overlap with existing VQA dataset. We further provide an in-depth analysis of 6 Med-MLLMs and compare them with 3 human specialists, providing insights into their competencies and limitations in various medical contexts. Our work not only advances the understanding of Med-MLLMs' capabilities but also sets a precedent for future evaluations and the safe deployment of these models in clinical environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑