多模态 GPT-4 视觉在医学中专家级准确率背后的隐藏缺陷
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
AI总结：
该研究评估了 GPT-4V 在 NEJM 影像挑战中的表现，发现其多选题准确率虽与医生相当，但在做出正确选择时仍有 35.5% 的病例存在推理依据缺陷，提示临床部署前需深入评估其推理过程。
摘要：
近期研究表明，具备视觉能力的生成式预训练 Transformer 4（GPT-4V）在医学挑战任务中表现优于人类医生。然而，这些评估主要仅关注多选题的准确率。我们的研究通过全面分析 GPT-4V 在解答《新英格兰医学杂志》（NEJM）影像挑战——一项旨在测试医学专业人员知识和诊断能力的影像测验——时的图像理解推理依据、医学知识回忆以及逐步多模态推理，扩展了当前的研究范围。评估结果证实，GPT-4V 在多选题准确率方面与人类医生表现相当（81.6% 对 77.8%）。在医生回答错误的病例中，GPT-4V 也表现良好，准确率超过 78%。然而，我们发现 GPT-4V 在做出正确最终选择的病例中频繁呈现有缺陷的推理依据（35.5%），其中图像理解方面最为突出（27.2%）。尽管 GPT-4V 在多选题中具有高准确率，我们的研究结果强调，在将此类多模态 AI 模型整合到临床工作流程之前，有必要对其推理依据进行更深入的评估。
英文摘要：
Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
