arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.11257cs.CLcs.AIcs.LG
收藏

谁是最佳侦探?LLM 与 ML 在检测不连贯的四年级数学答案中的比较

Who's the Best Detective? LLMs vs. MLs in Detecting Incoherent Fourth Grade Math Answers

Felipe Urrutia, Roberto Araya

AI总结:

本文比较 GPT-3、BLOOM、YOU 与传统 ML 分类器在检测四年级数学不连贯答案上的表现,发现 LLM 在不同少样本设置下均逊于 ML,尤其受递归问答和学生拼写错误困扰。

摘要:

开放式问题的书面答案相比多项选择题,可能对学习产生更长远的影响。然而,教师必须立即审阅这些答案,并要求学生重做那些不连贯的答案,这一点至关重要。对教师而言,这项任务可能既困难又耗时。一种可能的解决方案是自动检测不连贯答案;其中一个选择是使用大型语言模型(LLM)实现审阅自动化。本文使用三个 LLM——GPT-3、BLOOM 和 YOU——分析四年级学生的数学作答。我们在零样本、一样本、二样本、三样本和四样本设置下使用它们,并将其性能与使用机器学习(ML)训练的多种分类器结果进行比较。我们发现,在检测不连贯答案方面,LLM 的表现不如 ML。困难似乎在于同时包含问题与答案的递归式题目,以及带有典型四年级学生拼写错误的学生回答。进一步检查后,我们发现 ChatGPT 模型也面临同样的挑战。

英文摘要:

Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑