arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.00844cs.CLcs.AI
收藏

利用大型语言模型实现临床综述的论文自动筛选

Automated Paper Screening for Clinical Reviews Using Large Language Models

Cumming School of Medicine, University of Calgary(卡尔加里大学卡明医学院);Temerty Faculty of Medicine, University of Toronto(多伦多大学特默蒂医学院)

Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Mike Paget, Christopher Naugler

AI总结:

本研究提出基于OpenAI GPT API的临床综述论文自动筛选工作流,在6篇综述超24000条数据上实现0.91的准确率,可辅助提升临床综述效率与质量。

摘要:

目的:评估OpenAI GPT API在真实临床综述数据集中准确高效识别相关标题和摘要的性能,并将其性能与两名独立人类评审员标注的真值标签进行对比。方法:我们提出了一种利用OpenAI GPT API筛选临床综述中标题与摘要的新型工作流。我们编写了一个Python脚本,以自然语言形式的筛选标准,以及经至少两名人类评审员筛选过的标题与摘要数据集语料向GPT API发起调用。我们在6篇综述论文、超24000条标题与摘要的筛选任务中,对比了该模型与人类评审论文的性能。结果:我们的结果显示,模型准确率为0.91,排除论文的敏感度为0.91,纳入论文的敏感度为0.76。在随机选择的论文子集上,GPT API展现出为自身决策提供推理依据的能力;在被要求对部分错误分类的结果解释推理过程时,它能够修正最初的决策。结论:GPT API有望简化临床综述流程,为研究人员节省宝贵的时间与精力,并提升临床综述的整体质量。通过将该工作流置于辅助地位,作为研究人员与评审员的助手而非替代者,GPT API可提升效率,助力医学研究得出更准确可靠的结论。

英文摘要:

Objective: To assess the performance of the OpenAI GPT API in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review datasets and compare its performance against ground truth labelling by two independent human reviewers. Methods: We introduce a novel workflow using the OpenAI GPT API for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the GPT API with the screening criteria in natural language and a corpus of title and abstract datasets that have been filtered by a minimum of two human reviewers. We compared the performance of our model against human-reviewed papers across six review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. On a randomly selected subset of papers, the GPT API demonstrated the ability to provide reasoning for its decisions and corrected its initial decision upon being asked to explain its reasoning for a subset of incorrect classifications. Conclusion: The GPT API has the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, the GPT API can enhance efficiency and lead to more accurate and reliable conclusions in medical research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑