arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.14499cs.LGcs.CY
收藏

使用大型语言模型进行作业自动评分:来自生物信息学课程的见解

Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course

University of Ljubljana(卢布尔雅那大学);Faculty of Computer and Information Science(计算机与信息科学学院);Baylor College of Medicine(贝勒医学院)

Pavlin G. Poličar, Martin Špendl, Tomaž Curk, Blaž Zupan

AI总结:

本研究在生物信息学课程中评估了LLM用于作业自动评分的表现,发现精心设计的提示可使LLM达到与人类助教相当的评分准确性和反馈质量,且开源模型不逊于商业模型。

摘要:

通过作业为学生提供个性化反馈是教育的基础,支持他们的学习和发展。研究表明,及时、高质量的反馈在改善学习成果方面发挥着关键作用。然而,在拥有大量学生的课堂中大规模提供个性化反馈,由于所需的大量时间和精力,往往是不切实际的。自然语言处理和大型语言模型(LLMs)的最新进展提供了一种有前景的解决方案,能够高效地提供个性化反馈。这些技术可以减轻课程工作人员的工作量,同时提高学生满意度和学习成果。然而,它们的成功实施需要在真实课堂中进行彻底的评估和验证。我们展示了在卢布尔雅那大学2024/25学年《生物信息学导论》课程中,基于LLM的书面作业评分器的实际评估结果。在整个学期中,超过100名学生回答了36个基于文本的问题,其中大部分使用LLM自动评分。在一项盲研究中,学生在不知道反馈来源的情况下,同时收到来自LLM和人类助教的反馈,随后对反馈质量进行评分。我们对六种商业和开源LLM进行了系统评估,并将其评分表现与人类助教进行了比较。我们的结果表明,通过精心设计的提示,LLM可以实现与人类评分者相当的评分准确性和反馈质量。我们的结果还表明,开源LLM的表现与商业LLM一样好,使得学校能够在保持隐私的同时实施自己的评分系统。

英文摘要:

Providing students with individualized feedback through assignments is a cornerstone of education that supports their learning and development. Studies have shown that timely, high-quality feedback plays a critical role in improving learning outcomes. However, providing personalized feedback on a large scale in classes with large numbers of students is often impractical due to the significant time and effort required. Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback. These technologies can reduce the workload of course staff while improving student satisfaction and learning outcomes. Their successful implementation, however, requires thorough evaluation and validation in real classrooms. We present the results of a practical evaluation of LLM-based graders for written assignments in the 2024/25 iteration of the Introduction to Bioinformatics course at the University of Ljubljana. Over the course of the semester, more than 100 students answered 36 text-based questions, most of which were automatically graded using LLMs. In a blind study, students received feedback from both LLMs and human teaching assistants without knowing the source, and later rated the quality of the feedback. We conducted a systematic evaluation of six commercial and open-source LLMs and compared their grading performance with human teaching assistants. Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑