评估提示策略与大语言模型在系统性文献综述筛选中的应用：相关性与任务阶段分类
Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification
AI总结：
本研究评估六种LLM与五种提示策略在系统性文献综述筛选中的交互效果，发现CoT-少样本平衡最佳，并提出低成本模型初筛加边界案例升级的分阶段工作流。
摘要：
本研究量化了提示策略如何与大语言模型（LLM）相互作用，以实现系统性文献综述（SLR）筛选阶段的自动化。我们评估了六个LLM（GPT-4o、GPT-4o-mini、DeepSeek-Chat-V3、Gemini-2.5-Flash、Claude-3.5-Haiku、Llama-4-Maverick），在五种提示类型（零样本、少样本、思维链（CoT）、CoT-少样本、自我反思）下，跨相关性分类和六个二级任务进行评估，使用准确率、精确率、召回率和F1指标。结果显示出显著的模型-提示交互效应：CoT-少样本产生最可靠的精确率-召回率平衡；零样本在高灵敏度轮次中最大化召回率；自我反思因过度包容和跨模型不稳定性而表现不佳。GPT-4o和DeepSeek提供稳健的整体性能，而GPT-4o-mini以显著更低的美元成本表现出竞争力。针对相关性分类的成本-性能分析（每1000篇摘要）揭示了模型-提示组合之间存在巨大的绝对差异；GPT-4o-mini在各种提示下均保持低成本，且在GPT-4o-mini上使用结构化提示（CoT/CoT-少样本）能以较小的增量成本提供有吸引力的F1。我们推荐一种分阶段工作流：（1）部署带有结构化提示的低成本模型进行首轮筛选；（2）仅将边界案例升级到更高能力的模型。这些发现凸显了LLM在自动化文献筛选方面不均衡但充满前景的潜力。通过系统分析提示-模型交互，我们为任务自适应的LLM部署提供了比较基准和实践指导。
英文摘要：
This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought (CoT), CoT-few-shot, self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects: CoT-few-shot yields the most reliable precision-recall balance; zero-shot maximizes recall for high-sensitivity passes; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-mini offer attractive F1 at a small incremental cost. We recommend a staged workflow that (1) deploys low-cost models with structured prompts for first-pass screening and (2) escalates only borderline cases to higher-capacity models. These findings highlight LLMs' uneven but promising potential to automate literature screening. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
