arXiv 2606.066962026-10-06cs.CVcs.AI

MMBU: 大规模多模态生物医学理解基准，用于探测视觉语言模型的感知能力

MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

Stanford University（斯坦福大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Instituto Tecnológico de Monterrey（蒙特雷技术学院）；Monash University（墨尔本大学）；University of Cambridge（剑桥大学）；Shanghai Jiao Tong University（上海交通大学）；Shandong University（山东大学）

Ryan D'Cunha, Alejandro Lozano, Xiaoxiao Sun, Daniel Vela Jarquin, Min Woo Sun, Josiah Aklilu, James Burgess, Yuhui Zhang, Mark Endo, Ryan Nayebi, Akira Nishii,… 展开作者

Ryan D'Cunha, Alejandro Lozano, Xiaoxiao Sun, Daniel Vela Jarquin, Min Woo Sun, Josiah Aklilu, James Burgess, Yuhui Zhang, Mark Endo, Ryan Nayebi, Akira Nishii, Maximilian Rokuss, Paola Avila Robayo, Jin Ye, Ming Hu, Zhongying Deng, Junjun He, Xin Chen, Yue Yao, Robert Tibshirani, Jeffrey J. Nirschl, Serena Yeung-Levy

备注：: Expanded model results, revised evaluation description, and updated supplementary material

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MMBU基准，涵盖35个子模态，通过分类、定位和检测任务系统评估VLM在生物医学领域的视觉感知和泛化能力，发现高准确率可能掩盖感知缺陷。

摘要：

视觉和语言模型（VLM）在转变生物医学成像工作流程方面具有巨大潜力，从检测胸部X光片中的病变到显微镜下的细胞特征分析。然而，实现这一潜力需要稳健且细粒度的视觉感知。模型需要正确解释图像中的细微特征，并且必须在不同的生物医学模态、尺度和上下文中做到这一点。尽管如此，当前的基准仍然有限。为了解决这些差距，我们引入了大规模多模态生物医学理解（MMBU）基准。它是迄今为止最大的生物医学视觉和语言基准，涵盖35个子模态，具有丰富的结构化元数据。它包括开放和封闭版本的非接地分类、接地分类和物体检测，从而能够系统地评估模型在生物尺度、临床环境和成像模态上的性能。通过评估15个开源权重和2个前沿VLM，我们发现虽然医学适应为某些模型带来了可衡量的提升，但通常在高准确率报告中的表现可能掩盖了视觉感知和领域泛化方面的缺陷。

英文摘要：

Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy. Realizing this potential, however, requires robust and fine-grained visual perception. Models need to correctly interpret subtle features in images, and they must do so across diverse biomedical modalities, scales, and contexts. Nevertheless, current benchmarks remain limited. To address these gaps, we introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark. It is the largest biomedical vision and language benchmark to date, covering 35 submodalities with rich structured metadata. It includes both open and closed versions of ungrounded classification, grounded classification, and object detection, enabling systematic evaluation of model performance across biological scales, clinical settings, and imaging modalities. Evaluating 16 open-weight and 5 frontier VLMs in the main comparison, we find that while medical adaptation provides measurable gains for some models, the high accuracy often reported on established benchmarks can mask deficiencies in visual perception and domain generalization. We further define an open-ended MMBU hard set, split into a released public subset and a held-out private subset, to stress-test frontier models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。