重新评估用于系统综述的大型语言模型布尔查询生成
Reassessing Large Language Model Boolean Query Generation for Systematic Reviews
AI总结：
本研究系统重现了Wang等人和Staudinger等人关于LLMs辅助系统综述布尔查询生成的工作，通过解决输出格式限制和示例选择等被忽略因素，发现提示设计和模型选择是成功构建查询的关键驱动因素。
摘要：
系统综述是解决高度聚焦研究问题的全面文献综述，代表了医学领域的最高证据形式。该过程的一个关键步骤是开发复杂的布尔查询以检索相关文献。鉴于手动构建这些查询的难度，近期研究探索了大型语言模型（LLMs）来辅助其构建。首批研究之一Wang等人探讨了ChatGPT在此任务中的应用，随后Staudinger等人在一项可重复性研究中评估了多个LLMs。然而，后者忽略了原始工作的几个关键方面，包括(i)生成查询的验证，(ii)输出格式限制，以及(iii)思维链（Guided）提示的示例选择。因此，其发现与原始研究存在显著差异。在本研究中，我们在解决这些被忽略因素的同时，系统地重现了这两项研究。结果表明，查询有效性在不同模型和提示设计间差异显著，引导式查询构建得益于精心挑选的种子研究。总体而言，提示设计和模型选择是成功构建查询的关键驱动因素。我们的发现为LLMs在布尔查询生成中的潜力提供了更清晰的理解，并强调了针对模型和提示进行特定优化的重要性。系统综述的复杂性增加了方法开发与重现的挑战，但也凸显了该领域可重复性研究的重要性。
英文摘要：
Systematic reviews are comprehensive literature reviews that address highly focused research questions and represent the highest form of evidence in medicine. A critical step in this process is the development of complex Boolean queries to retrieve relevant literature. Given the difficulty of manually constructing these queries, recent efforts have explored Large Language Models (LLMs) to assist in their formulation. One of the first studies,Wang et al., investigated ChatGPT for this task, followed by Staudinger et al., which evaluated multiple LLMs in a reproducibility study. However, the latter overlooked several key aspects of the original work, including (i) validation of generated queries, (ii) output formatting constraints, and (iii) selection of examples for chain-of-thought (Guided) prompting. As a result, its findings diverged significantly from the original study. In this work, we systematically reproduce both studies while addressing these overlooked factors. Our results show that query effectiveness varies significantly across models and prompt designs, with guided query formulation benefiting from well-chosen seed studies. Overall, prompt design and model selection are key drivers of successful query formulation. Our findings provide a clearer understanding of LLMs' potential in Boolean query generation and highlight the importance of model- and prompt-specific optimisations. The complex nature of systematic reviews adds to challenges in both developing and reproducing methods but also highlights the importance of reproducibility studies in this domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
