临床推理的幻觉：一个基准揭示了视觉语言模型在临床能力方面的广泛差距
The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency
AI总结：
本研究提出B&J基准测试，揭示了视觉语言模型在临床推理任务中存在显著的多模态整合能力不足问题，强调需在多模态整合和视觉理解上取得突破才能实现临床应用。
摘要：
背景：基础模型在临床实践和公共卫生中的快速整合，需要对其真正的临床推理能力进行严格评估，而不仅仅是狭窄的考试成功。当前的基准测试通常基于医学执业考试或精心编写的场景，无法捕捉到真实世界患者护理中所需的综合、多模态推理。方法：我们开发了Bones and Joints（B&J）基准测试，这是一个包含1245个问题的综合评估框架，这些问题源自骨科和运动医学中的真实患者案例。该基准测试评估模型在7项任务上的表现，这些任务模拟了临床推理路径，包括知识回忆、文本和图像解读、诊断生成、治疗计划和理由提供。我们评估了11个视觉语言模型（VLMs）和6个大型语言模型（LLMs），并将其表现与专家推导的地面真实值进行比较。结果：我们的结果表明，任务类型之间存在显著的性能差距。虽然最先进的模型在结构化多选题上实现了高准确率，超过90%，但在需要多模态整合的开放性任务上，其表现明显下降，准确率 barely 接近60%。VLMs在解读医学图像方面表现出显著的局限性，并且经常表现出严重的文本驱动幻觉，经常忽略矛盾的视觉证据。值得注意的是，专门针对医学应用进行微调的模型在一般模型上没有一致的优势。结论：当前的人工智能模型尚未在复杂的多模态推理中具备临床能力。其安全部署目前应限制在支持性的文本基础角色中。未来在核心临床任务上的进步需要在多模态整合和视觉理解方面取得根本性的突破。
英文摘要：
Background: The rapid integration of foundation models into clinical practice and public health necessitates a rigorous evaluation of their true clinical reasoning capabilities beyond narrow examination success. Current benchmarks, typically based on medical licensing exams or curated vignettes, fail to capture the integrated, multimodal reasoning essential for real-world patient care. Methods: We developed the Bones and Joints (B&J) Benchmark, a comprehensive evaluation framework comprising 1,245 questions derived from real-world patient cases in orthopedics and sports medicine. This benchmark assesses models across 7 tasks that mirror the clinical reasoning pathway, including knowledge recall, text and image interpretation, diagnosis generation, treatment planning, and rationale provision. We evaluated eleven vision-language models (VLMs) and six large language models (LLMs), comparing their performance against expert-derived ground truth. Results: Our results demonstrate a pronounced performance gap between task types. While state-of-the-art models achieved high accuracy, exceeding 90%, on structured multiple-choice questions, their performance markedly declined on open-ended tasks requiring multimodal integration, with accuracy scarcely reaching 60%. VLMs demonstrated substantial limitations in interpreting medical images and frequently exhibited severe text-driven hallucinations, often ignoring contradictory visual evidence. Notably, models specifically fine-tuned for medical applications showed no consistent advantage over general-purpose counterparts. Conclusions: Current artificial intelligence models are not yet clinically competent for complex, multimodal reasoning. Their safe deployment should currently be limited to supportive, text-based roles. Future advancement in core clinical tasks awaits fundamental breakthroughs in multimodal integration and visual understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
