MedBookVQA：源自开放获取书籍的系统且全面的医学基准
MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book
AI总结：
MedBookVQA 从开放获取医学教科书中构建包含 5,000 个临床问题的多模态基准，通过分层标注评估多种 MLLMs，揭示 GMAI 能力缺口并推动临床 AI 评估发展。
摘要：
在多模态大语言模型（MLLMs）的推动下，通用医学人工智能（GMAI）的加速发展，为解决持续的医疗挑战（包括人力短缺和不断上升的成本）提供了变革性潜力。系统性评估基准的并行开发成为实现性能评估和提供技术指导的关键要务。与此同时，作为宝贵的知识来源，医学教科书在基准开发方面的潜力仍未得到充分利用。在此，我们提出 MedBookVQA，一个源自开放获取医学教科书的系统且全面的多模态基准。为构建该基准，我们提出了一套标准化流程，用于自动提取医学图像，并将其与相应的医学叙述进行上下文对齐。基于这些整理的数据，我们生成了 5,000 个临床相关问题，涵盖模态识别、疾病分类、解剖结构识别、症状诊断和外科手术。一个多层标注系统通过涵盖医学成像模态（42 个类别）、身体解剖结构（125 个结构）和临床专科（31 个科室）的分层分类法对查询进行分类，从而能够对医学子领域进行细致分析。我们评估了广泛的 MLLMs，包括专有模型、开源模型、医学模型和推理模型，揭示了不同任务类型和模型类别之间存在显著性能差异。我们的发现突显了当前 GMAI 系统的关键能力缺口，同时确立了源自教科书的医学多模态基准作为重要评估工具的地位。MedBookVQA 将源自教科书的基准测试确立为推进临床 AI 的关键范式，既暴露了 GMAI 系统的局限性，又提供了跨专科的解剖结构化性能指标。
英文摘要：
The accelerating development of general medical artificial intelligence (GMAI), powered by multimodal large language models (MLLMs), offers transformative potential for addressing persistent healthcare challenges, including workforce deficits and escalating costs. The parallel development of systematic evaluation benchmarks emerges as a critical imperative to enable performance assessment and provide technological guidance. Meanwhile, as an invaluable knowledge source, the potential of medical textbooks for benchmark development remains underexploited. Here, we present MedBookVQA, a systematic and comprehensive multimodal benchmark derived from open-access medical textbooks. To curate this benchmark, we propose a standardized pipeline for automated extraction of medical figures while contextually aligning them with corresponding medical narratives. Based on this curated data, we generate 5,000 clinically relevant questions spanning modality recognition, disease classification, anatomical identification, symptom diagnosis, and surgical procedures. A multi-tier annotation system categorizes queries through hierarchical taxonomies encompassing medical imaging modalities (42 categories), body anatomies (125 structures), and clinical specialties (31 departments), enabling nuanced analysis across medical subdomains. We evaluate a wide array of MLLMs, including proprietary, open-sourced, medical, and reasoning models, revealing significant performance disparities across task types and model categories. Our findings highlight critical capability gaps in current GMAI systems while establishing textbook-derived multimodal benchmarks as essential evaluation tools. MedBookVQA establishes textbook-derived benchmarking as a critical paradigm for advancing clinical AI, exposing limitations in GMAI systems while providing anatomically structured performance metrics across specialties.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
