arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03361eess.IVcs.CV

GMAI-MMBench：面向通用医疗人工智能的综合多模态评估基准

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Shanghai AI Laboratory（上海人工智能实验室）；University of Washington（华盛顿大学）；Monash University（莫纳什大学）；East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Cambridge（剑桥大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, J… 展开作者

Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 GMAI-MMBench，一个覆盖 284 个数据集、38 种图像模态、18 类任务和 4 种感知粒度的通用医疗 AI 多模态评估基准，并通过对 50 个 LVLMs 的评估揭示现有模型在医疗应用中的关键不足。

摘要：

大型视觉语言模型（LVLMs）能够处理成像、文本和生理信号等多种数据类型，并可应用于多个领域。在医学领域，LVLMs 具有为诊断和治疗提供实质性帮助的巨大潜力。在此之前，开发用于评估 LVLMs 在各种医学应用中有效性的基准至关重要。当前的基准通常建立在特定学术文献之上，主要集中于单一领域，并且缺乏不同的感知粒度。因此，它们面临特定挑战，包括临床相关性有限、评估不完整以及对交互式 LVLMs 的指导不足。为了解决这些局限性，我们开发了 GMAI-MMBench，这是迄今为止最全面的通用医疗人工智能基准，具有良好分类的数据结构和多感知粒度。它以视觉问答（VQA）格式构建，来自 284 个数据集，涵盖 38 种医学图像模态、18 个临床相关任务、18 个科室和 4 种感知粒度。此外，我们实现了一种词法树结构，允许用户自定义评估任务，适应各种评估需求，并有力支持医学人工智能研究和应用。我们评估了 50 个 LVLMs，结果表明，即使是先进的 GPT-4o 也仅达到 53.96% 的准确率，表明仍有很大的改进空间。此外，我们确定了当前前沿 LVLMs 中五个关键不足，需要解决这些问题以推动更好的医学应用发展。我们相信 GMAI-MMBench 将激励社区构建面向通用医疗人工智能的下一代 LVLMs。

英文摘要：

Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑