一种用于教育调查反馈分析的大型语言模型方法
A Large Language Model Approach to Educational Survey Feedback Analysis
AI总结：
本文评估GPT-4和GPT-3.5在教育反馈调查分析中的潜力，通过零样本NLP任务序列实现人类水平表现，并验证LLM在提取教育调查见解中的有效性。
摘要：
本文评估了大型语言模型（LLMs）GPT-4和GPT-3.5在教育反馈调查中辅助提取见解的潜力。教育领域对LLM用例的探索主要集中在教学上，对教育反馈分析能力的探索较少。教育中的调查分析涉及寻找课程差距或评估教师等目标，通常需要耗时的文本回复人工处理。LLMs有望提供一种灵活的方式来实现这些目标，而无需专门的机器学习模型或微调。我们通过将这些目标视为自然语言处理（NLP）任务序列，展示了一种实现这些目标的通用方法，包括分类（多标签、多类和二分类）、提取、主题分析和情感分析，每项任务均由LLM执行。我们将这些工作流程应用于包含2500条生物医学科学课程期末调查评论的真实世界数据集，并在所有任务中评估了零样本方法（即无需示例或标注训练数据），这反映了教育环境中标注数据通常稀缺的现状。通过应用有效的提示实践，我们使用GPT-4在多项任务上实现了人类水平的表现，从而支持实现典型目标所需的工作流程。我们还展示了检查LLM思维链（CoT）推理的潜力，这可提供有助于在实践中增强信心的见解。此外，本研究开发了一套通用的分类类别，适用于各种课程类型（在线、混合或面对面），并支持自定义。我们的结果表明，LLMs可用于从调查文本中提取一系列见解。
英文摘要：
This paper assesses the potential for the large language models (LLMs) GPT-4 and GPT-3.5 to aid in deriving insight from education feedback surveys. Exploration of LLM use cases in education has focused on teaching and learning, with less exploration of capabilities in education feedback analysis. Survey analysis in education involves goals such as finding gaps in curricula or evaluating teachers, often requiring time-consuming manual processing of textual responses. LLMs have the potential to provide a flexible means of achieving these goals without specialized machine learning models or fine-tuning. We demonstrate a versatile approach to such goals by treating them as sequences of natural language processing (NLP) tasks including classification (multi-label, multi-class, and binary), extraction, thematic analysis, and sentiment analysis, each performed by LLM. We apply these workflows to a real-world dataset of 2500 end-of-course survey comments from biomedical science courses, and evaluate a zero-shot approach (i.e., requiring no examples or labeled training data) across all tasks, reflecting education settings, where labeled data is often scarce. By applying effective prompting practices, we achieve human-level performance on multiple tasks with GPT-4, enabling workflows necessary to achieve typical goals. We also show the potential of inspecting LLMs' chain-of-thought (CoT) reasoning for providing insight that may foster confidence in practice. Moreover, this study features development of a versatile set of classification categories, suitable for various course types (online, hybrid, or in-person) and amenable to customization. Our results suggest that LLMs can be used to derive a range of insights from survey text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
