Beyond the Hype: A dispassionate look at vision-language models in medical scenario
Bioengineering Department and Imperial-X, Imperial College London(生物工程部门和Imperial-X,帝国理工学院伦敦分校);GSK, Artificial Intelligence and Machine Learning(GSK,人工智能与机器学习);Cardiovascular Research Centre, Royal Brompton Hospital(心血管研究中心,皇家布里托尼医院);School of Biomedical Engineering and Imaging Sciences, King’s College London(生物医学工程与成像科学学院,国王学院伦敦分校)
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across diverse tasks, garnering significant attention in AI communities. However, their performance and reliability in specialized domains such as medicine remain insufficiently assessed. In particular, most assessments over-concentrate on evaluating VLMs based on simple Visual Question Answering (VQA) on multi-modality data, while ignoring the in-depth characteristics of LVLMs. In this study, we introduce RadVUQA, a novel Radiological Visual Understanding and Question Answering benchmark, to comprehensively evaluate existing LVLMs. RadVUQA mainly validates LVLMs across five dimensions: 1) Anatomical understanding, assessing the models' ability to visually identify biological structures; 2) Multimodal comprehension, which involves the capability of interpreting linguistic and visual instructions to produce desired outcomes; 3) Quantitative and spatial reasoning, evaluating the models' spatial awareness and proficiency in combining quantitative analysis with visual and linguistic information; 4) Physiological knowledge, measuring the models' capability to comprehend functions and mechanisms of organs and systems; and 5) Robustness, which assesses the models' capabilities against unharmonized and synthetic data. The results indicate that both generalized LVLMs and medical-specific LVLMs have critical deficiencies with weak multimodal comprehension and quantitative reasoning capabilities. Our findings reveal the large gap between existing LVLMs and clinicians, highlighting the urgent need for more robust and intelligent LVLMs. The code is available at https://github.com/Nandayang/RadVUQA
MedDoc-Bot: A Chat Tool for Comparative Analysis of Large Language Models in the Context of the Pediatric Hypertension Guideline
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School(布伦瑞克工业大学与汉诺威医学院Peter L. Reichertz医学信息学研究所);Hannover Medical School(汉诺威医学院)
Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups
Department of Computer Science and Engineering, University of Notre Dame(计算机科学与工程系,诺丁汉大学);Department of Electrical Engineering, University of Notre Dame(电气工程系,诺丁汉大学);Weill Cornell Medicine(韦尔医学院);Department of Computer Science, Emory University(计算机科学系,埃默里大学)