High-performance automated abstract screening with large language model ensembles
University of Oxford(牛津大学);Oxford University Hospitals NHS Foundation Trust(牛津大学医院国民保健服务基金会信托);University of Cambridge(剑桥大学);Georgetown University(乔治城大学);MedStar Washington Hospital Center(MedStar华盛顿医院中心);King’s College London(伦敦国王学院);University College London(伦敦大学学院)
Rohan Sanghera, Arun James Thirunavukarasu, Marc El Khoury, Jessica O'Logbon, Yuqing Chen, Archie Watt, Mustafa Mahmood, Hamid Butt, George Nishimura, Andrew Soltan
Large language models (LLMs) excel in tasks requiring processing and interpretation of input text. Abstract screening is a labour-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies identified by a literature search. Here, LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialled on systematic reviews in a full issue of the Cochrane Library to evaluate their accuracy in zero-shot binary classification for abstract screening. Trials over a subset of 800 records identified optimal prompting strategies and demonstrated superior performance of LLMs to human researchers in terms of sensitivity (LLM-max = 1.000, human-max = 0.775), precision (LLM-max = 0.927, human-max = 0.911), and balanced accuracy (LLM-max = 0.904, human-max = 0.865). The best performing LLM-prompt combinations were trialled across every replicated search result (n = 119,691), and exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096). 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458, with less observed performance drop in larger trials. Significant variation in performance was observed between reviews, highlighting the importance of domain-specific validation before deployment. LLMs may reduce the human labour cost of systematic review with maintained or improved accuracy and sensitivity. Systematic review is the foundation of evidence synthesis across academic disciplines, including evidence-based medicine, and LLMs may increase the efficiency and quality of this mode of research.
Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach
Duke University(杜克大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Duke University School of Medicine(杜克大学医学院);Duke Clinical Research Institute(杜克临床研究所)
IHC-LLMiner: Automated extraction of tumour immunohistochemical profiles from PubMed abstracts using large language models
Institute of Health Informatics, University College London(伦敦大学学院健康信息学研究所);Research Department of Pathology, University College London(伦敦大学学院病理学研究部);Department of Pathology, Hadassah-Hebrew University Medical Center(哈达萨-希伯来大学医学中心病理学系);Department of Cellular Pathology, University College London Hospitals NHS Foundation Trust(伦敦大学学院医院NHS信托基金会细胞病理学系);School of Health & Wellbeing, University of Glasgow(格拉斯哥大学健康与福祉学院)
NeuraHealth: An Automated Screening Pipeline to Detect Undiagnosed Cognitive Impairment in Electronic Health Records with Deep Learning and Natural Language Processing
Massachusetts General Hospital(麻省总医院);McCance Center for Brain Health(麦坎斯大脑健康中心)
痴呆相关的认知障碍(CI)是一种神经退行性疾病,影响全球超过5500万人,并以每3秒一个新病例的速度快速增长。全球75%的病例未被诊断,其中90%发生在低收入和中等收入国家,估计全球每年的成本为1.3万亿美元,预计到2030年将达到2.8万亿美元。由于无法治愈,临床试验反复失败,缺乏早期诊断,死亡率为100%。电子健康记录(EHR)中的信息可以为CI的早期检测提供重要线索,但专家的手动审查是繁琐且容易出错的。已经提出了几种计算方法,然而,他们缺乏对复杂的语言结构的EHR的语言上下文的增强的理解。因此,我提出了一个新的,更准确的框架,神经健康,以确定谁没有早期诊断的患者。在NeuraHealth中,我使用来自Mass General Brigham BioBank的患者EHR,微调了一个基于双向注意力的深度学习自然语言处理模型来对序列进行分类。序列预测用于生成结构化特征作为患者水平正则化逻辑回归模型的输入。这种两步框架创建了高维度,优于所有现有的最先进的计算方法以及临床方法。此外,我将这些模型集成到一个现实世界的产品,一个网络应用程序,以创建一个自动化的EHR筛查管道,用于可扩展和高速发现EHR中未检测到的CI,使早期诊断在医疗设施和医疗服务稀缺的地区变得可行。
Evaluating Spoken Language as a Biomarker for Automated Screening of Cognitive Impairment
Imperial College London(帝国理工学院);UK Dementia Research Institute(英国痴呆症研究所);Imperial College Healthcare NHS Trust(帝国理工学院医疗保健国民保健服务信托基金);University of Surrey(萨里大学);University of Southern California(南加州大学)