IMCBench：面向多模态大语言模型在图像基础医疗对话中的基准测试
IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
AI总结：
提出IMCBench基准，结合临床图像与合成患者档案模拟多轮医疗对话，从安全性、准确性和不确定性使用三个维度评估模型，发现准确描述不等于安全指导。
摘要：
近期大型语言模型和视觉-语言模型的进展使得对多模态数据进行推理成为可能，为决策支持和分诊等临床应用提供了机会。然而，现有的医学AI基准测试是碎片化的：有些支持多轮对话但缺乏图像，而另一些提供多模态输入但专注于单轮问答任务。为填补这一空白，我们引入了IMCBench，一个基于图像的多轮医疗对话基准测试，它将真实的、公开可用的临床图像与合成患者档案配对，以模拟真实的患者-临床医生互动。每个对话在三个临床维度上评估：安全性、准确性和诊断中不确定性的适当使用。我们使用LLM-as-Jury评分方法，针对四个模型家族（Claude、GPT、Nova和Llama）的八个多模态前沿模型进行基准测试，每个模型在1-5分制上评分，并与专家临床医生注释进行校准。我们的结果显示，Claude Opus 4.6获得了最高总分（3.61），其次是Claude Sonnet 4.6（3.30）和GPT-5.2（3.29），尽管没有模型在所有维度上占主导地位，并且对于恶性和罕见疾病，安全性都会下降（Δ = -0.27）。消融研究进一步揭示，视觉输入和EHR上下文都有助于安全指导（当两者分别移除时，安全性平均下降0.18和0.23），更强的模型更有效地利用视觉特征。这些发现共同表明，准确的临床描述并不能保证安全的患者指导，这激发了在医学AI中需要多维度评估框架的需求。
英文摘要：
Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions ($Δ$ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
