Systematic reviews are time-consuming endeavors that require knowledgeable human reviewers to screen studies for relevance and extract data following a specific coding scheme before any analysis or synthesis can occur. Large language models (LLMs) hold promise for substantially accelerating this process and reducing reviewer workload, yet their application within the context of systematic reviews in the field of education remains underexplored. We address this issue in two ways: through empirical studies and the iterative development of an open-source software tool. First, we conducted two empirical studies examining the efficacy of using LLMs for data extraction using data from a published review on pedagogical agents. We extracted a variety of data types from 112 studies and compared the results to data extracted by human coding. Results indicate that LLMs struggled with extracting data accurately and therefore are not ready to be used as primary data extraction tools without explicit human validation of the data extracted. These findings highlight the dire need for a human-in-the-loop (HIL) approach to AI-assisted data extraction. We then propose a HIL workflow and introduce and describe the development of a free, web-based, open-source tool designed to support user-friendly, human-validated data extraction with LLMs.
LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education
The Hong Kong University of Science and Technology(香港科技大学);Korea Advanced Institute of Science and Technology(韩国科学技术院);Lingnan University(岭南大学);Texas A&M University(德克萨斯农工大学)
Learning behavior accounts for background-related advantage in AI-assisted education
Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);Microsoft AI Asia(微软亚洲人工智能研究院);Microsoft Research Cambridge(微软剑桥研究院);Paul and Marcia Center on Contemporary China, Princeton University(普林斯顿大学当代中国保罗和玛西娅中心);Center for Social Research, Guanghua School of Management, Peking University(北京大学光华管理学院社会研究中心);Microsoft Research Asia(微软亚洲研究院);University of Science and Technology of China(中国科学技术大学)
Data Comics for Education: Evaluating Effectiveness, Benefits, and the Ethics of AI-Assisted Creation
Faculty of Information Technology, Monash University(蒙纳士大学信息技术学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院);Department of Applied Social Sciences, The Hong Kong Polytechnic University(香港理工大学应用社会科学系)
The Role of Instructional Guidance in Generative AI-Assisted Learning: Empirical Evidence from Construction Engineering Education
Dept. of Civil, Environmental, and Geospatial Engineering, Michigan Technological Univ.(土木、环境与地理空间工程系,密歇根技术大学);Dept. of Civil and Construction Engineering, Western Michigan Univ.(土木与建设工程系,西部密歇根大学);Dept. of Psychology and Human Factors, Michigan Technological Univ.(心理学与人因工程系,密歇根技术大学)
LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations
University of Siegen(锡根大学);Institute for Knowledge-Based Systems and Knowledge Management(知识与知识管理研究所);Institute for Embedded Systems(嵌入式系统研究所);Department of Digital Health Sciences and Biomedicine(数字健康科学与生物医学系)
Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale
National Tutoring Observatory(国家辅导观察站);FreshCognate;Cornell University(康奈尔大学);Massachusetts Institute of Technology(麻省理工学院);Carnegie Mellon University(卡内基梅隆大学)