利用LLMs实现系统文献综述中的半自动语料筛选
Leveraging LLMs for Semi-Automatic Corpus Filtration in Systematic Literature Reviews
AI总结:
本文提出利用多个大型语言模型实现系统文献综述中的半自动语料筛选,通过人类监督和交互式控制提高效率和准确性。
摘要:
系统文献综述(SLR)的创建对于分析研究领域的现状并指导未来研究方向至关重要。然而,检索和过滤SLR所需的文献语料过程非常耗时,需要大量人工努力,因为数字图书馆中的关键词搜索往往返回大量无关的出版物。在本工作中,我们提出了一种利用多个大型语言模型(LLMs)的管道,通过描述性提示对论文进行分类,并通过共识方案共同决定。整个过程由人类监督并通过我们的开源可视化分析网页界面LLMSurver进行交互式控制,该界面使能够实时检查和修改模型输出。我们使用来自最近一个包含超过8000篇候选论文的SLR的地面真实数据来评估我们的方法,基准测试了中2024年和2025年秋季的开放和商业最先进的LLMs。结果表明,我们的管道显著减少了人工努力,同时比单个人类标注者实现了更低的错误率。此外,现代开源模型证明对于这项任务是足够的,使该方法变得可访问且成本效益高。总体而言,我们的工作展示了负责任的人机协作如何在学术流程中加速和增强系统文献综述。
英文摘要:
The creation of systematic literature reviews (SLR) is critical for analyzing the landscape of a research field and guiding future research directions. However, retrieving and filtering the literature corpus for an SLR is highly time-consuming and requires extensive manual effort, as keyword-based searches in digital libraries often return numerous irrelevant publications. In this work, we propose a pipeline leveraging multiple large language models (LLMs), classifying papers based on descriptive prompts and deciding jointly using a consensus scheme. The entire process is human-supervised and interactively controlled via our open-source visual analytics web interface, LLMSurver, which enables real-time inspection and modification of model outputs. We evaluate our approach using ground-truth data from a recent SLR comprising over 8,000 candidate papers, benchmarking both open and commercial state-of-the-art LLMs from mid-2024 and fall 2025. Results demonstrate that our pipeline significantly reduces manual effort while achieving lower error rates than single human annotators. Furthermore, modern open-source models prove sufficient for this task, making the method accessible and cost-effective. Overall, our work demonstrates how responsible human-AI collaboration can accelerate and enhance systematic literature reviews within academic workflows.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。