arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.08152cs.CL
收藏

使用 ChatGPT 或 GPT-4 等大型语言模型作为临床助手的潜力与陷阱

The Potential and Pitfalls of using a Large Language Model such as ChatGPT or GPT-4 as a Clinical Assistant

Pangaea Data Limited(泛大陆数据有限公司);Data Science Institute, Imperial College London(帝国理工学院数据科学研究所);Hong Kong University of Science and Technology(香港科技大学)

Jingqing Zhang, Kai Sun, Akshay Jagadeesh, Mahta Ghahfarokhi, Deepa Gupta, Ashok Gupta, Vibhor Gupta, Yike Guo

AI总结:

本文通过真实世界 EHR 患者识别和假设病例前瞻性诊断两项实验,评估 ChatGPT/GPT-4 的临床辅助能力,发现 GPT-4 表现优异但存在错误、过度医疗与隐私风险。

摘要:

近期研究表明,ChatGPT 和 GPT-4 在多项医学领域任务上展现出令人期待的性能。然而,尚无研究利用大规模真实世界电子健康记录数据库评估其性能,也没有评估其在覆盖全部疾病表现类型的患者中提供临床诊断辅助的效用。我们使用 ChatGPT 和 GPT-4 开展了两项分析:一项基于真实世界大型电子健康记录数据库识别具有特定医学诊断的患者,另一项则在对假设患者进行前瞻性评估时,为医护人员提供诊断辅助。结果显示,在疾病分类任务中,结合思维链与少样本提示,GPT-4 的 F1 分数最高可达 96%。在患者评估方面,GPT-4 四次中能有三次作出准确诊断。但也出现了事实性错误陈述、忽视关键医学发现、建议不必要检查以及过度治疗等问题。这些问题连同隐私方面的担忧,使得这些模型目前尚不足以用于真实世界临床场景。不过,与配置传统机器学习工作流相比,其所需数据和提示工程时间有限,凸显出它们在各类医疗应用中实现规模化扩展的潜力。

英文摘要:

Recent studies have demonstrated promising performance of ChatGPT and GPT-4 on several medical domain tasks. However, none have assessed its performance using a large-scale real-world electronic health record database, nor have evaluated its utility in providing clinical diagnostic assistance for patients across a full range of disease presentation. We performed two analyses using ChatGPT and GPT-4, one to identify patients with specific medical diagnoses using a real-world large electronic health record database and the other, in providing diagnostic assistance to healthcare workers in the prospective evaluation of hypothetical patients. Our results show that GPT-4 across disease classification tasks with chain of thought and few-shot prompting can achieve performance as high as 96% F1 scores. For patient assessment, GPT-4 can accurately diagnose three out of four times. However, there were mentions of factually incorrect statements, overlooking crucial medical findings, recommendations for unnecessary investigations and overtreatment. These issues coupled with privacy concerns, make these models currently inadequate for real world clinical use. However, limited data and time needed for prompt engineering in comparison to configuration of conventional machine learning workflows highlight their potential for scalability across healthcare applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑