arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.11840cs.HC
收藏

教育领域系统综述的AI辅助数据提取

AI-Assisted Data Extraction for Systematic Reviews in Education

University of Florida(佛罗里达大学);Wright State University(赖特州立大学)

Noah L. Schroeder, Chris Davis Jaldi, Shan Zhang, Kweku Yamoah

AI总结:

本研究针对教育领域系统综述中LLMs应用不足的问题,通过两项实证研究发现LLMs数据提取准确性不足,提出人在回路工作流程并开发相关开源工具,为AI辅助数据提取提供可行方案。

摘要:

系统综述是耗时的工作,需要专业的人工评审员按照特定编码方案筛选研究相关性并提取数据,才能开展分析或综合。大型语言模型(LLMs)有望大幅加速该过程并减轻评审员工作量,但其在教育领域系统综述中的应用仍未得到充分探索。我们通过实证研究和迭代开发开源软件工具两种方式解决该问题。首先,我们开展两项实证研究,利用已发表的关于教学智能体的综述数据,检验LLMs用于数据提取的效果:从112项研究中提取多种数据类型,并与人工编码提取的数据进行比较。结果表明,LLMs在准确提取数据方面存在困难,因此在未对提取的数据进行明确人工验证的情况下,暂不能作为主要数据提取工具使用。这些发现凸显了对人在回路(HIL)的AI辅助数据提取方法的迫切需求。随后,我们提出一种HIL工作流程,并介绍和描述一款免费、基于网页的开源工具的开发情况,该工具旨在支持使用LLMs进行用户友好、经人工验证的数据提取。

英文摘要:

Systematic reviews are time-consuming endeavors that require knowledgeable human reviewers to screen studies for relevance and extract data following a specific coding scheme before any analysis or synthesis can occur. Large language models (LLMs) hold promise for substantially accelerating this process and reducing reviewer workload, yet their application within the context of systematic reviews in the field of education remains underexplored. We address this issue in two ways: through empirical studies and the iterative development of an open-source software tool. First, we conducted two empirical studies examining the efficacy of using LLMs for data extraction using data from a published review on pedagogical agents. We extracted a variety of data types from 112 studies and compared the results to data extracted by human coding. Results indicate that LLMs struggled with extracting data accurately and therefore are not ready to be used as primary data extraction tools without explicit human validation of the data extracted. These findings highlight the dire need for a human-in-the-loop (HIL) approach to AI-assisted data extraction. We then propose a HIL workflow and introduce and describe the development of a free, web-based, open-source tool designed to support user-friendly, human-validated data extraction with LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑