arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.13375cs.CLcs.AI
收藏

GPT-4在医学挑战性问题上的能力

Capabilities of GPT-4 on Medical Challenge Problems

Microsoft(微软);OpenAI(开放人工智能研究中心)

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, Eric Horvitz

AI总结:

本文全面评估了通用大模型GPT-4在USMLE和MultiMedQA等医学考试与基准上的表现,发现其无需专门提示即超过及格线20分以上,优于GPT-3.5和医学微调模型Med-PaLM,且概率校准更好,并展示了其在医学推理、个性化教学和反事实情景构建方面的能力。

摘要:

大型语言模型(LLMs)在包括医学在内的多个领域的自然语言理解和生成方面展现出了卓越的能力。我们对GPT-4这一最先进的大型语言模型在医学能力考试和基准数据集上的表现进行了全面评估。GPT-4是一个通用模型,并未通过训练专门针对医学问题,也未被设计用于解决临床任务。我们的分析涵盖了两套USMLE(美国医师执照考试)的官方练习材料,USMLE是一个用于评估临床能力并在美国授予行医执照的三步考试项目。我们还评估了其在MultiMedQA基准数据集套件上的表现。除了衡量模型性能外,我们还进行了实验,以研究同时包含文本和图像的测试题目对模型性能的影响,探查训练过程中对内容的记忆情况,并研究概率校准——这在医学等高风险应用中至关重要。我们的结果表明,GPT-4在没有任何专门提示设计的情况下,在USMLE上的成绩超过及格线20分以上,并且优于早期的通用模型(GPT-3.5)以及在医学知识上专门微调的模型(Med-PaLM,即Flan-PaLM 540B的提示调优版本)。此外,GPT-4的校准度显著优于GPT-3.5,显示出其在预测自身答案正确可能性方面的能力有了大幅提升。我们还通过一项案例研究对模型行为进行了定性探索,该案例展示了GPT-4解释医学推理、为学生个性化解释以及围绕医学案例交互式构建新的反事实情景的能力。我们讨论了这些发现对GPT-4在医学教育、评估和临床实践中潜在应用的意义,并适当关注了准确性和安全性方面的挑战。

英文摘要:

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑