arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.09338cs.CL
收藏

预训练大语言模型GPT-4在自动短答案评分中的表现

Performance of the Pre-Trained Large Language Model GPT-4 on Automated Short Answer Grading

ETH Zurich(苏黎世联邦理工学院)

Gerd Kortemeyer

AI总结:

本文评估了通用预训练大语言模型GPT-4在自动短答案评分任务上的表现,发现其性能与手工设计模型相当,但弱于经过专门训练的预训练大语言模型。

摘要:

自动短答案评分(ASAG)十多年来一直是机器学习研究中的一个活跃领域。它承诺让教育工作者在人类评分者数量有限的情况下,仍能对大规模招生课程中的自由形式回答进行评分并给出反馈。多年来,经过精心训练的模型已取得越来越高的性能。最近,预训练大语言模型(LLMs)成为一种通用商品,一个引人关注的问题是:一个无需额外训练的通用工具与专用模型相比表现如何。我们研究了GPT-4在标准基准2-way和3-way数据集SciEntsBank和Beetle上的表现,其中除了标准任务——对学生答案与参考答案的一致性进行评分外,我们还考察了不提供参考答案的情况。我们发现,总体而言,预训练的通用GPT-4大语言模型的性能与手工设计的模型相当,但不如经过专门训练的预训练大语言模型。

英文摘要:

Automated Short Answer Grading (ASAG) has been an active area of machine-learning research for over a decade. It promises to let educators grade and give feedback on free-form responses in large-enrollment courses in spite of limited availability of human graders. Over the years, carefully trained models have achieved increasingly higher levels of performance. More recently, pre-trained Large Language Models (LLMs) emerged as a commodity, and an intriguing question is how a general-purpose tool without additional training compares to specialized models. We studied the performance of GPT-4 on the standard benchmark 2-way and 3-way datasets SciEntsBank and Beetle, where in addition to the standard task of grading the alignment of the student answer with a reference answer, we also investigated withholding the reference answer. We found that overall, the performance of the pre-trained general-purpose GPT-4 LLM is comparable to hand-engineered models, but worse than pre-trained LLMs that had specialized training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑