AISysRev -- 基于 LLM 的标题-摘要筛选工具
AISysRev -- LLM-based Tool for Title-abstract Screening
AI总结:
AISysRev 是一个基于大语言模型的标题-摘要筛选工具,通过容器化Web应用实现自动化筛选,支持多种LLM模型,减少文献评估负担,但需人类干预处理边界情况。
摘要:
系统综述的开展是繁重的。在筛选或研究选择阶段,论文数量可能令人望而却步。最近的研究表明,大语言模型(LLMs)可以执行标题-摘要筛选并支持人类完成该任务。为此,我们开发了AISysRev,一个基于LLM的筛选工具,作为容器化Web应用实现。该工具接受包含论文标题和摘要的CSV文件。用户指定包含和排除标准。可以使用多种不同的LLM,如Gemini、Claude、Mistral或ChatGPT via OpenRouter。我们还支持本地托管模型和任何兼容OpenAI SDK的模型。AISysRev实现了零样本和少样本提示,并允许通过界面显示LLM结果,以指导人类审阅员。LLM调用被并行化,意味着筛选速度通常在每分钟100到300篇论文之间,取决于模型和主机。为了展示该工具在实际中的使用,我们用137篇论文进行了定性试验研究。我们的发现表明,论文可以分为四类:容易包含、容易排除、边界包含和边界排除。边界情况,其中LLM容易出错,突显了需要人类干预的必要性。尽管LLM不取代系统综述中的人类判断,但它们可以减少评估大量科学文献的负担。
英文摘要:
Conducting systematic reviews is laborious. In the screening or study selection phase, the number of papers can be overwhelming. Recent research has demonstrated that large language models (LLMs) can perform title-abstract screening and support humans in the task. To this end, we developed AISysRev, an LLM-based screening tool implemented as a containerized web application. The tool accepts CSV files containing paper titles and abstracts. Users specify inclusion and exclusion criteria. Multiple different LLMs can be used, such as Gemini, Claude, Mistral or ChatGPT via OpenRouter. We also support locally hosted models and any model compatible with the OpenAI SDK. AISysRev implements both zero-shot and few-shot prompting, and also allows for manual screening through interfaces that display LLM results as guidance for human reviewers. LLM calls are parallelized, meaning screening speed is typically between 100 to 300 papers per minute, depending on the model and the host. To demonstrate the tool's use in practice, we conducted a qualitative trial study with 137 papers using the tool. Our findings indicate that papers can be classified into four categories: Easy Includes, Easy Excludes, Boundary Includes, and Boundary Excludes. The Boundary cases, where LLMs are prone to errors, highlight the need for human intervention. While LLMs do not replace human judgment in systematic reviews, they can reduce the burden of assessing large volumes of scientific literature. Video: https://www.youtube.com/watch?v=HeblemlgnAQ Tool: https://github.com/EvoTestOps/AISysRev
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。