arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.02451cs.CLcs.DLcs.IR
收藏

基于大语言模型集成的高性能自动化摘要筛选

High-performance automated abstract screening with large language model ensembles

University of Oxford(牛津大学);Oxford University Hospitals NHS Foundation Trust(牛津大学医院国民保健服务基金会信托);University of Cambridge(剑桥大学);Georgetown University(乔治城大学);MedStar Washington Hospital Center(MedStar华盛顿医院中心);King’s College London(伦敦国王学院);University College London(伦敦大学学院)

Rohan Sanghera, Arun James Thirunavukarasu, Marc El Khoury, Jessica O'Logbon, Yuqing Chen, Archie Watt, Mustafa Mahmood, Hamid Butt, George Nishimura, Andrew Soltan

AI总结:

本文评估了六种大语言模型在Cochrane Library系统综述摘要筛选中的零样本分类性能,发现LLMs在灵敏度、精确率和平衡准确率上优于人类,并通过LLM-人类及LLM-LLM集成实现完美灵敏度,表明LLMs可降低系统综述人力成本并提升效率与质量。

摘要:

大语言模型(LLMs)在需要处理与解释输入文本的任务中表现出色。摘要筛选是系统综述中劳动密集型的组成部分,涉及对文献检索所识别的大量研究反复应用纳入与排除标准。本研究在Cochrane Library一整期系统综述上对LLMs(GPT-3.5 Turbo、GPT-4 Turbo、GPT-4o、Llama 3 70B、Gemini 1.5 Pro和Claude Sonnet 3.5)进行了试验,以评估其在摘要筛选零样本二元分类中的准确性。在800条记录的子集上进行的试验确定了最优提示策略,并证明LLMs在灵敏度(LLM最大值=1.000,人类最大值=0.775)、精确率(LLM最大值=0.927,人类最大值=0.911)和平衡准确率(LLM最大值=0.904,人类最大值=0.865)方面均优于人类研究人员。表现最佳的LLM-提示组合在每个复制的检索结果(n=119,691)上进行了试验,表现出稳定的灵敏度(范围0.756-1.000),但精确率有所下降(范围0.004-0.096)。66个LLM-人类和LLM-LLM集成表现出完美的灵敏度,最大精确率为0.458,且在较大规模试验中观察到的性能下降更少。不同综述之间观察到显著的性能差异,凸显了在部署前进行领域特定验证的重要性。LLMs可以在保持或提高准确性和灵敏度的情况下降低系统综述的人力劳动成本。系统综述是跨学科领域(包括循证医学)证据综合的基础,LLMs可能提高这种研究模式的效率和质量。

英文摘要:

Large language models (LLMs) excel in tasks requiring processing and interpretation of input text. Abstract screening is a labour-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies identified by a literature search. Here, LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialled on systematic reviews in a full issue of the Cochrane Library to evaluate their accuracy in zero-shot binary classification for abstract screening. Trials over a subset of 800 records identified optimal prompting strategies and demonstrated superior performance of LLMs to human researchers in terms of sensitivity (LLM-max = 1.000, human-max = 0.775), precision (LLM-max = 0.927, human-max = 0.911), and balanced accuracy (LLM-max = 0.904, human-max = 0.865). The best performing LLM-prompt combinations were trialled across every replicated search result (n = 119,691), and exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096). 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458, with less observed performance drop in larger trials. Significant variation in performance was observed between reviews, highlighting the importance of domain-specific validation before deployment. LLMs may reduce the human labour cost of systematic review with maintained or improved accuracy and sensitivity. Systematic review is the foundation of evidence synthesis across academic disciplines, including evidence-based medicine, and LLMs may increase the efficiency and quality of this mode of research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑