Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in various multimodal tasks. However, their potential in the medical domain remains largely unexplored. A significant challenge arises from the scarcity of diverse medical images spanning various modalities and anatomical regions, which is essential in real-world medical applications. To solve this problem, in this paper, we introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. This benchmark is collected from 73 different medical datasets, including 12 different modalities and covering more than 20 distinct anatomical regions. Importantly, all images in this benchmark are sourced from authentic medical scenarios, ensuring alignment with the requirements of the medical field and suitability for evaluating LVLMs. Through our extensive experiments, we have found that existing LVLMs struggle to address these medical VQA problems effectively. Moreover, what surprises us is that medical-specialized LVLMs even exhibit inferior performance to those general-domain models, calling for a more versatile and robust LVLM in the biomedical field. The evaluation results not only reveal the current limitations of LVLM in understanding real medical images but also highlight our dataset's significance. Our code with dataset are available at https://github.com/OpenGVLab/Multi-Modality-Arena.
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Shanghai AI Laboratory（上海人工智能实验室）；University of Washington（华盛顿大学）；Monash University（莫纳什大学）；East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Cambridge（剑桥大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；College of AI, Tsinghua University（清华大学人工智能学院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；Cognition and Intelligent Technology Laboratory（认知与智能技术实验室）
Medical Large Vision Language Models with Multi-Image Visual Ability
Dept. of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China（计算机科学与工程系，香港中文大学，香港，中国）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China（医学智能与XR研究所，香港中文大学，香港，中国）
Beyond the Hype: A dispassionate look at vision-language models in medical scenario
Bioengineering Department and Imperial-X, Imperial College London（生物工程部门和Imperial-X，帝国理工学院伦敦分校）；GSK, Artificial Intelligence and Machine Learning（GSK，人工智能与机器学习）；Cardiovascular Research Centre, Royal Brompton Hospital（心血管研究中心，皇家布里托尼医院）；School of Biomedical Engineering and Imaging Sciences, King’s College London（生物医学工程与成像科学学院，国王学院伦敦分校）
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Department of Electrical and Computer Engineering, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院机器人与机电工程系）；Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院智能机器人系）；Vector Institute（向量研究所）；Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)（釜山科学技术大学计算机科学与工程系）
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
School of Intelligent Science and Technology, Nanjing University（南京大学智能科学与技术学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学医疗数据科学国家研究院）；School of Biomedical Engineering, Nanjing University（南京大学生物医学工程学院）
Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Department of computer science, University of Central Florida, Orlando, USA（计算机科学系，中央佛罗里达大学）；Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA（生物工程系，宾夕法尼亚大学）；Department of electrical engineering, Columbia university, New York, NY, USA（电气工程系，哥伦比亚大学）；Technical University of Applied Sciences Regensburg, Regensburg, Germany（应用科学技术大学（雷根斯堡））；Department of Surgery, University of Calgary, Calgary, Alberta, Canada（外科系，卡尔加里大学）；University College of Nabi Akram, Tabriz, Iran（纳比阿克兰大学）；School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran（电气工程学院，伊朗科学技术大学）
MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcement
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Renmin University of China（中国人民大学）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine（埃默里大学医学院放射肿瘤学系与温希普癌症研究所）；Department of Biostatistics and Bioinformatics, Emory University（埃默里大学生物统计学与生物信息学系）
U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding
Fudan University（复旦大学）；University of Oxford（牛津大学）；Dolphin AI；Beihang University（北京航空航天大学）；Hong Kong Baptist University（香港浸会大学）；University of Cambridge（剑桥大学）；The Chinese University of Hong Kong（香港中文大学）；Esonic Imaging；Technical University of Denmark（丹麦技术大学）
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
York University（约克大学）；Vector Institute（向量研究所）；University of British Columbia（不列颠哥伦比亚大学）；University of Toronto（多伦多大学）；Unity Health Toronto / St. Michael’s Hospital（多伦多联合健康/圣迈克尔医院）；University Health Network（大学健康网络）；Arc Institute（弧研究所）；Queen's University（女王大学）
Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
Korea University（韩国大学）；AIGEN Sciences（AIGEN公司）；Hanyang University College of Medicine（翰林大学医学院）；Asan Medical Center, University of Ulsan College of Medicine（釜山大学医学院阿桑医院）；Yale University（耶鲁大学）
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Universiti Malaya（马来亚大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港智能系统研究所人工智能与机器人中心）；University College London（伦敦大学学院）
GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis
The Hong Kong Polytechnic University（香港理工大学）；National University of Singapore（新加坡国立大学）；Sichuan University（四川大学）；West China Hospital of Sichuan University（四川大学华西医院）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）
LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models
Yale University（耶鲁大学）；Imperial College London（伦敦帝国学院）；Australian National University（澳大利亚国立大学）；University of Georgia（佐治亚大学）；National University of Singapore（新加坡国立大学）；RMIT University（皇家墨尔本理工大学）
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Alibaba（阿里巴巴）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）
Towards Statistical Factuality Guarantee for Large Vision-Language Models
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Intuit（英图伊特公司）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Intuit AI Research（英图伊特人工智能研究院）
Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Cognition and Intelligent Technology Laboratory (CIT Lab), Institute of Meta-Medical, Fudan University（复旦大学元医学研究院认知与智能技术实验室）；Engineering Research Center of AI and Robotics, Ministry of Education, China（中国教育部人工智能与机器人工程研究中心）；Jilin Provincial Key Laboratory of Intelligence Science and Engineering（吉林省智能科学与工程重点实验室）
SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
ROSE Lab, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore（南洋理工大学罗思实验室，跨学科研究生项目，新加坡）；ROSE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学罗思实验室，电子与电气工程学院，新加坡）；City University of Hong Kong, Hong Kong SAR（香港城市大学，香港特别行政区）；Shanghai Jiao Tong University, China（上海交通大学，中国）；Singapore University of Technology and Design, Singapore（新加坡科技设计大学，新加坡）；VinUniversity, Hanoi, Vietnam（越南文大学，河内，越南）
Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis
Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；The Hong Kong University of Science and Technology (GZ)（香港科学与技术大学）；National University of Singapore（新加坡国立大学）；CVTE；Sun Yat-sen University（孙中山大学）；Department of Diagnostic Radiology, The University of Hong Kong（香港大学放射科）；Imaging and Interventional Radiology, Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院影像与介入放射科）；School of Computer Science, Peking University（北京大学计算机科学系）
