arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10965cs.CLcs.CYcs.LG
收藏

医疗人工智能中的泛化能力:临床大语言模型的评估

Generalization in Healthcare AI: Evaluation of a Clinical Large Language Model

New York University(纽约大学);NYU Langone Health(纽约大学朗格尼健康中心)

Salman Rahman, Lavender Yao Jiang, Saadia Gabriel, Yindalon Aphinyanaphongs, Eric Karl Oermann, Rumi Chunara

AI总结:

本研究通过评估基于临床记录训练的 ClinicLLM 在 30 天再入院预测中的泛化表现,揭示了样本量、年龄、合并症和记录字数等关键影响因素,并证明医院特异性局部微调能最有效提升泛化能力。

摘要:

大语言模型(LLMs)的进展为医疗保健领域带来了新的机遇,可用于改善患者护理、临床决策以及提升医生和管理人员的工作流程。然而,这些模型的潜力在很大程度上取决于它们能否在不同临床环境和患者群体中有效泛化,而这一挑战在早期开发中往往被低估。为了更好地理解这些挑战的原因并为缓解方法提供依据,我们评估了 ClinicLLM,这是一个基于 [HOSPITAL] 临床记录训练的大语言模型,重点分析其在 30 天全因再入院预测任务上的表现,并关注不同医院和患者特征之间的变异性。我们发现,在样本较少的医院、使用政府保险和未明确保险类型的患者、老年人以及合并症较多的患者中,泛化能力尤其较差。为了理解泛化不足的原因,我们考察了微调样本量、记录内容(每条记录的字数)、患者特征(合并症水平、年龄、保险类型、行政区)以及卫生系统特征(医院、30 天全因再入院率和死亡率)。我们使用描述性统计和有监督分类来识别相关特征。我们发现,除样本量外,患者年龄、合并症数量以及记录字数都是与泛化相关的重要因素。最后,我们比较了局部微调(针对特定医院)、基于实例增强的微调和基于聚类的微调在改善泛化方面的效果。其中,局部微调被证明最为有效,AUC 提升了 0.25% 至 11.74%(在数据有限的环境中帮助最大)。总体而言,本研究为在医疗保健这一具有重要社会意义的领域中改进大语言模型的部署,以及提升其在更广泛人群中的表现提供了新的见解。

英文摘要:

Advances in large language models (LLMs) provide new opportunities in healthcare for improved patient care, clinical decision-making, and enhancement of physician and administrator workflows. However, the potential of these models importantly depends on their ability to generalize effectively across clinical environments and populations, a challenge often underestimated in early development. To better understand reasons for these challenges and inform mitigation approaches, we evaluated ClinicLLM, an LLM trained on [HOSPITAL]'s clinical notes, analyzing its performance on 30-day all-cause readmission prediction focusing on variability across hospitals and patient characteristics. We found poorer generalization particularly in hospitals with fewer samples, among patients with government and unspecified insurance, the elderly, and those with high comorbidities. To understand reasons for lack of generalization, we investigated sample sizes for fine-tuning, note content (number of words per note), patient characteristics (comorbidity level, age, insurance type, borough), and health system aspects (hospital, all-cause 30-day readmission, and mortality rates). We used descriptive statistics and supervised classification to identify features. We found that, along with sample size, patient age, number of comorbidities, and the number of words in notes are all important factors related to generalization. Finally, we compared local fine-tuning (hospital specific), instance-based augmented fine-tuning and cluster-based fine-tuning for improving generalization. Among these, local fine-tuning proved most effective, increasing AUC by 0.25% to 11.74% (most helpful in settings with limited data). Overall, this study provides new insights for enhancing the deployment of large language models in the societally important domain of healthcare, and improving their performance for broader populations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑