Advances in large language models (LLMs) provide new opportunities in healthcare for improved patient care, clinical decision-making, and enhancement of physician and administrator workflows. However, the potential of these models importantly depends on their ability to generalize effectively across clinical environments and populations, a challenge often underestimated in early development. To better understand reasons for these challenges and inform mitigation approaches, we evaluated ClinicLLM, an LLM trained on [HOSPITAL]'s clinical notes, analyzing its performance on 30-day all-cause readmission prediction focusing on variability across hospitals and patient characteristics. We found poorer generalization particularly in hospitals with fewer samples, among patients with government and unspecified insurance, the elderly, and those with high comorbidities. To understand reasons for lack of generalization, we investigated sample sizes for fine-tuning, note content (number of words per note), patient characteristics (comorbidity level, age, insurance type, borough), and health system aspects (hospital, all-cause 30-day readmission, and mortality rates). We used descriptive statistics and supervised classification to identify features. We found that, along with sample size, patient age, number of comorbidities, and the number of words in notes are all important factors related to generalization. Finally, we compared local fine-tuning (hospital specific), instance-based augmented fine-tuning and cluster-based fine-tuning for improving generalization. Among these, local fine-tuning proved most effective, increasing AUC by 0.25% to 11.74% (most helpful in settings with limited data). Overall, this study provides new insights for enhancing the deployment of large language models in the societally important domain of healthcare, and improving their performance for broader populations.
Gender Bias in Large Language Models for Healthcare: Assignment Consistency and Clinical Implications
Duke-NUS Medical School(杜克-新加坡国立大学医学院);Singapore General Hospital(新加坡中央医院);University of Antwerp(安特卫普大学);Duke University School of Medicine(杜克大学医学院);SingHealth(新加坡保健服务集团);Singapore National Eye Centre(新加坡国家眼科中心);Weill Cornell Medicine(威尔康奈尔医学院);Mass General Brigham(麻省总医院布里格姆);Harvard Medical School(哈佛医学院);National University of Singapore(新加坡国立大学)
University of Toronto(多伦多大学);Vector Institute for AI(人工智能向量研究所);Johns Hopkins Medical Institutions(约翰霍普金斯医学机构);University Health Network(医疗网络);Algoma University(阿尔戈马大学);University of Iowa Hospitals & Clinics(爱荷华大学医院与诊所);University of Edinburgh(爱丁堡大学)
A Study of Generative Large Language Model for Medical Research and Healthcare
University of Florida(佛罗里达大学);University of Florida Health Cancer Center(佛罗里达大学健康癌症中心);NVIDIA(英伟达);UF Clinical and Translational Science Institute(佛罗里达大学临床与转化科学研究所)
Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
Peking University(北京大学);Peking University Cancer Hospital and Institute(北京大学肿瘤医院暨北京市肿瘤防治研究所);National Engineering Laboratory for Big Data Analysis and Applications, Peking University(北京大学大数据分析与应用技术国家工程实验室);Peking University Changsha Institute for Computing and Digital Economy(北京大学长沙计算与数字经济研究院)
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University(京都大学);The Hong Kong University of Science and Technology(香港科技大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);The University of Tokyo(东京大学)
DeepSeek in Healthcare: A Survey of Capabilities, Risks, and Clinical Applications of Open-Source Large Language Models
Weill Cornell Medicine, Cornell University(威尔康奈尔医学院,康奈尔大学);Johns Hopkins University(约翰斯·霍普金斯大学);Touro University College of Osteopathic Medicine(图罗大学整骨医学院)
A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises
Emory University(埃默里大学);University of Michigan(密歇根大学);Georgia Institute of Technology(佐治亚理工学院);Rensselaer Polytechnic Institute(伦斯勒理工学院);Case Western Reserve University(凯斯西储大学);Iowa State University(爱荷华州立大学);Weill Cornell Medicine, Cornell University(康奈尔大学威尔医学院)
Efficient and Effective Internal Memory Retrieval for LLM-Based Healthcare Prediction
University of Connecticut(康涅狄格大学);University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校);UMass Chan Medical School(马萨诸塞大学陈医学院)