arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10147cs.SEcs.AI
收藏

在系统映射研究中使用大型语言模型以支持其开展:一名实践者视角的简要报告

On the Use of a Large Language Model to Support the Conduction of a Systematic Mapping Study: A Brief Report from a Practitioner's View

Federal University of Goiás\ Institute Goiânia Goiás Brazil;Pontifical Catholic University of Rio de Janeiro\ of Informatics Rio de Janeiro Rio de Janeiro Brazil;Boston University\ of Computer Science Boston Massachusetts USA;Federal University of Goiás\ Institute;Pontifical Catholic University of Rio de Janeiro\ of Informatics;Boston University\ of Computer Science

Cauã Ferreira Barros, Marcos Kalinowski, Mohamad Kassab, Valdemar Vicente Graciano Neto

AI总结:

本文从实践者视角报告了使用大型语言模型开展系统映射研究的经验,讨论了其在时间节省和数据标准化方面的优势,以及构建提示、幻觉和手动验证等挑战。

摘要:

大型语言模型(LLMs)的使用在科学界引起了越来越多的兴趣。LLMs能够处理大量文本数据并支持证据综合方法。尽管最近的研究强调了LLMs在系统综述中加速筛选和数据提取步骤的潜力,但对其在整个过程中实际应用的详细报告仍然很少。本文呈现了一项在LLMs支持下开展系统映射研究的经验报告,描述了所遵循的步骤、必要的调整以及面临的主要挑战。讨论了积极方面,如(i)重复任务中时间的显著减少,以及(ii)数据提取中的更大标准化。同时,也提到了消极方面,包括(i)构建可靠且结构良好的提示需要大量努力,特别是对于经验较少的用户,因为实现有效的提示可能需要多次迭代和测试,这可能会部分抵消预期的时间节省,(ii)幻觉的出现,以及(iii)需要持续的手动验证。作为贡献,这项工作为有兴趣在系统映射和综述中采用LLMs的研究人员提供了教训和实用建议,突出了效率提升以及需要考虑的方法论风险和限制。

英文摘要:

The use of Large Language Models (LLMs) has drawn growing interest within the scientific community. LLMs can handle large volumes of textual data and support methods for evidence synthesis. Although recent studies highlight the potential of LLMs to accelerate screening and data extraction steps in systematic reviews, detailed reports of their practical application throughout the entire process remain scarce. This paper presents an experience report on the conduction of a systematic mapping study with the support of LLMs, describing the steps followed, the necessary adjustments, and the main challenges faced. Positive aspects are discussed, such as (i) the significant reduction of time in repetitive tasks and (ii) greater standardization in data extraction, as well as negative aspects, including (i) considerable effort to build reliable well-structured prompts, especially for less experienced users, since achieving effective prompts may require several iterations and testing, which can partially offset the expected time savings, (ii) the occurrence of hallucinations, and (iii) the need for constant manual verification. As a contribution, this work offers lessons learned and practical recommendations for researchers interested in adopting LLMs in systematic mappings and reviews, highlighting both efficiency gains and methodological risks and limitations to be considered.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑