arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08704cs.CVcs.AI
收藏

超越炒作:冷静审视医学场景中的视觉语言模型

Beyond the Hype: A dispassionate look at vision-language models in medical scenario

Bioengineering Department and Imperial-X, Imperial College London(生物工程部门和Imperial-X,帝国理工学院伦敦分校);GSK, Artificial Intelligence and Machine Learning(GSK,人工智能与机器学习);Cardiovascular Research Centre, Royal Brompton Hospital(心血管研究中心,皇家布里托尼医院);School of Biomedical Engineering and Imaging Sciences, King’s College London(生物医学工程与成像科学学院,国王学院伦敦分校)

Yang Nan, Huichi Zhou, Xiaodan Xing, Guang Yang

AI总结:

研究提出RadVUQA基准,从解剖学理解、多模态理解、定量空间推理、生理学知识和鲁棒性五个维度全面评估医学场景下的LVLMs,发现现有模型在多模态理解和定量推理上存在严重缺陷,与临床医生水平差距巨大。

摘要:

大型视觉语言模型(LVLMs)的最新进展展示了其在各种任务上的卓越能力,引起了AI社区的极大关注。然而,它们在医学等专业领域的性能和可靠性仍未得到充分评估。特别是,大多数评估过度集中于基于多模态数据的简单视觉问答(VQA)来评估VLMs,而忽略了LVLMs的深层特性。在本研究中,我们引入了RadVUQA,一个新颖的放射学视觉理解与问答基准,以全面评估现有的LVLMs。RadVUQA主要从五个维度验证LVLMs:1) 解剖学理解,评估模型视觉识别生物结构的能力;2) 多模态理解,涉及解释语言和视觉指令以产生预期结果的能力;3) 定量和空间推理,评估模型的空间意识以及将定量分析与视觉和语言信息结合的能力;4) 生理学知识,衡量模型理解器官和系统功能及机制的能力;5) 鲁棒性,评估模型对未协调和合成数据的能力。结果表明,通用LVLMs和医疗专用LVLMs都存在关键缺陷,多模态理解和定量推理能力较弱。我们的发现揭示了现有LVLMs与临床医生之间的巨大差距,突出了对更强大、更智能的LVLMs的迫切需求。代码可在https://github.com/Nandayang/RadVUQA获取。

英文摘要:

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across diverse tasks, garnering significant attention in AI communities. However, their performance and reliability in specialized domains such as medicine remain insufficiently assessed. In particular, most assessments over-concentrate on evaluating VLMs based on simple Visual Question Answering (VQA) on multi-modality data, while ignoring the in-depth characteristics of LVLMs. In this study, we introduce RadVUQA, a novel Radiological Visual Understanding and Question Answering benchmark, to comprehensively evaluate existing LVLMs. RadVUQA mainly validates LVLMs across five dimensions: 1) Anatomical understanding, assessing the models' ability to visually identify biological structures; 2) Multimodal comprehension, which involves the capability of interpreting linguistic and visual instructions to produce desired outcomes; 3) Quantitative and spatial reasoning, evaluating the models' spatial awareness and proficiency in combining quantitative analysis with visual and linguistic information; 4) Physiological knowledge, measuring the models' capability to comprehend functions and mechanisms of organs and systems; and 5) Robustness, which assesses the models' capabilities against unharmonized and synthetic data. The results indicate that both generalized LVLMs and medical-specific LVLMs have critical deficiencies with weak multimodal comprehension and quantitative reasoning capabilities. Our findings reveal the large gap between existing LVLMs and clinicians, highlighting the urgent need for more robust and intelligent LVLMs. The code is available at https://github.com/Nandayang/RadVUQA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑