“我理解我为什么得到这个分数”:带反馈的自动简答题评分
"I understand why I got this grade": Automatic Short Answer Grading with Feedback
AI总结:
本文提出EngSAF数据集及LASFG策略,用于带反馈的自动简答题评分,Mistral-7B模型在未见答案和问题上准确率达75.4%和58.7%,并在真实考试中验证了系统有效性。
摘要:
近年来,利用人工智能(AI)自动化学生评估在教育领域引起了越来越多的兴趣。在不同类型的评估中,总结性评估在评价学生对课程的理解水平方面起着至关重要的作用。此类考试通常涉及简答题。然而,手动大规模地批改这些回答并提供有意义的反馈既耗时又费力。反馈尤其重要,因为它能帮助学生认识到自己的优势和需要改进的领域。尽管这项任务很重要,但支持带反馈生成的自动简答题评分的公开数据集却严重缺乏。为解决这一空白,我们引入了工程简答题反馈(EngSAF),这是一个专为带反馈的自动简答题评分设计的数据集。该数据集涵盖了多个工程领域的多样化主题、问题和答案模式,包含约5.8k个数据点。我们利用最先进的大语言模型(LLM)的生成能力,通过我们的标签感知合成反馈生成(LASFG)策略将反馈纳入数据集。本文强调了在实际教育环境中增强反馈的重要性,概述了数据集标注和反馈生成过程,对EngSAF进行了全面分析,并提供了基于不同LLM的零样本和微调基线以供未来比较。性能最佳的模型(Mistral-7B)在未见答案和未见问题测试集上分别达到了75.4%和58.7%的总体准确率。此外,我们通过在一所知名机构的真实期末考试的部署,展示了我们ASAG系统的效率和有效性。
英文摘要:
In recent years, there has been a growing interest in using Artificial Intelligence (AI) to automate student assessment in education. Among different types of assessments, summative assessments play a crucial role in evaluating a student's understanding level of a course. Such examinations often involve short-answer questions. However, grading these responses and providing meaningful feedback manually at scale is both time-consuming and labor-intensive. Feedback is particularly important, as it helps students recognize their strengths and areas for improvement. Despite the importance of this task, there is a significant lack of publicly available datasets that support automatic short-answer grading with feedback generation. To address this gap, we introduce Engineering Short Answer Feedback (EngSAF), a dataset designed for automatic short-answer grading with feedback. The dataset covers a diverse range of subjects, questions, and answer patterns from multiple engineering domains and contains ~5.8k data points. We incorporate feedback into our dataset by leveraging the generative capabilities of state-of-the-art large language models (LLMs) using our Label-Aware Synthetic Feedback Generation (LASFG) strategy. This paper underscores the importance of enhanced feedback in practical educational settings, outlines dataset annotation and feedback generation processes, conducts a thorough EngSAF analysis, and provides different LLMs-based zero-shot and finetuned baselines for future comparison. The best-performing model (Mistral-7B) achieves an overall accuracy of 75.4% and 58.7% on unseen answers and unseen question test sets, respectively. Additionally, we demonstrate the efficiency and effectiveness of our ASAG system through its deployment in a real-world end-semester exam at a reputed institute.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。