Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.
Mayo Clinic(梅奥诊所);the First Affiliated Hospital of Nanchang University(南昌大学第一附属医院);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);Eye Institute of Xiamen University(厦门大学眼科研究所)
DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4
The University of Georgia(佐治亚大学);Lehigh University(莱恩大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital and Harvard Medical School(麻省总医院和哈佛医学院);Mayo Clinic(梅奥诊所);ShanghaiTech University(上海科技大学);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney(悉尼大学);Tencent AI Lab, Tencent(腾讯AI实验室);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Wollongong(伍伦贡大学);The University of Adelaide(阿德莱德大学)
GP-GPT: Large Language Model for Gene-Phenotype Mapping
Department of Computer Science and Engineering, The University of Texas at Arlington(计算机科学与工程系,德克萨斯理工大学 Arlington 分校);School of Computing, University of Georgia(计算学院,佐治亚大学);Department of Computer Science, Indiana University Indianapolis(计算机科学系,印第安纳大学 Indianapolis 分校);Department of Radiology, Massachusetts General Hospital and Harvard Medical School(放射科,麻省总医院和哈佛医学院);Department of Statistics, University of Georgia(统计系,佐治亚大学);Department of Epidemiology & Biostatistics, University of Georgia(流行病学与生物统计学系,佐治亚大学);Department of Environmental Health Science, University of Georgia(环境流行病学系,佐治亚大学)
SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)
University of Texas at Austin(德克萨斯大学奥斯汀分校);Pontifical Catholic University of Rio Grande do Sul(南里奥格兰德天主教大学);University of Pittsburgh(匹兹堡大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Pennsylvania(宾夕法尼亚大学);University of Texas Southwestern(德克萨斯大学西南医学中心);University of Virginia(弗吉尼亚大学);Temple University(天普大学);Weill Cornell Medicine(威尔康奈尔医学院)
GPT-4 passes most of the 297 written Polish Board Certification Examinations
Adam Mickiewicz University(亚当·密茨凯维奇大学);Poznań University of Medical Sciences(波兹南医科大学);Nicolaus Copernicus University, Collegium Medicum(尼古拉·哥白尼大学医学院);Kazimierz Wielki University(卡齐米日·维尔基大学);The Society for the Substitution Treatment of Addiction “Medically Assisted Recovery”(药物辅助康复成瘾替代治疗协会)
Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
University of Guelph(圭尔夫大学);University of North Texas(北德克萨斯大学);Iowa State University(爱荷华州立大学);Zayed University(扎耶德大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);National University of Computer & Emerging Sciences(国立计算机与新兴科学大学)
TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation
Sichuan University(四川大学);Case Western Reserve University(凯斯西储大学);Beijing University of Posts and Telecommunications(北京邮电大学);Urgench State University(乌尔根奇国立大学);Gansu Agricultural University(甘肃农业大学)
IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment
University of Science and Technology of China(中国科学技术大学);School of Artificial Intelligence and Big Data, Hefei University(合肥大学人工智能与大数据学院);Anhui University of Science and Technology(安徽理工大学)
An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
Faculty of Computing and Information Technology, Sohar University(苏哈尔大学计算与信息技术学院);Foundation University School of Sciences and Technology(基础大学科学与技术学院);College of Science and Engineering, Hamad Bin Khalifa University(哈马德·本·哈利法大学科学与工程学院)
Evaluating multiple large language models in pediatric ophthalmology
Mayo Clinic(梅奥诊所);the First Affiliated Hospital of Nanchang University(南昌大学第一附属医院);Eye& ENT Hospital of Fudan University(复旦大学附属眼耳鼻喉科医院);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学)
3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models
Northwestern Polytechnical University(西北工业大学);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院);Clinical Research Center for Medical Imaging in Hunan Province(湖南省医学影像临床研究中心);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Georgia(佐治亚大学)
A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing
Department of Computing;Games, Teesside University, Middlesbrough, United Kingdom Centre for Digital Innovation, Teesside University, Middlesbrough, United Kingdom Faculty of Business \& Technology, University of Sunderland, Sunderland, United Kingdom
Shanghai East Hospital(上海东方医院);Tongji University School of Medicine(同济大学医学院);Institute of Ethnology and Anthropology(民族学与人类学研究所);Chinese Academy of Social Sciences(中国社会科学院)
ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulation
University of Texas at Austin(得克萨斯大学奥斯汀分校);Ulsan National Institute of Science and Technology(蔚山国家科学技术研究院);Korea Institute of Science and Technology (KIST) Europe(韩国科学技术研究院欧洲分院)
Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);University College London(伦敦大学学院);University of Cambridge(剑桥大学);LanBridge Group(兰桥集团);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究院)