arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.11902cs.CL
收藏

Evaluating the Knowledge Dependency of Questions(问题的知识依赖性评估)

Evaluating the Knowledge Dependency of Questions

Riiid AI Research(Riiid人工智能研究院);NAVER AI Lab(NAVER人工智能实验室);UC Berkeley(加州大学伯克利分校)

Hyeongdon Moon, Yoonseok Yang, Jamin Shin, Hangyeol Yu, Seunghyun Lee, Myeongho Jeong, Juneyoung Park, Minsam Kim, Seungtaek Choi

AI总结:

针对现有MCQ生成评估指标忽略教育价值的问题,提出知识依赖可答性(KDA)指标及两种自动近似指标,验证了其与人工KDA、课堂可用性的强相关性及结合传统指标后的质量预测能力。

摘要:

选择题(MCQ)自动生成有望大幅减少教育工作者在学生评估上投入的时间。然而,现有的MCQ生成评估指标(如BLEU、ROUGE、METEOR)聚焦于生成的MCQ与数据集中金标准样本的基于n-gram的相似度,忽略了其教育价值,无法评估MCQ考查学生对对应目标知识点掌握情况的能力。为解决该问题,我们提出一种新型自动评估指标,命名为知识依赖可答性(KDA),用于衡量在掌握目标知识点的前提下MCQ的可答性。具体而言,我们首先展示了如何基于人工调研的学生作答数据测量KDA;随后提出KDA_disc和KDA_cont两种自动评估指标,通过预训练语言模型模拟学生解题行为来近似KDA。通过人工研究我们发现,KDA_disc与KDA_soft和两类指标均强相关:一是KDA,二是专家标注的实际课堂场景可用性。此外,当与基于n-gram的相似度指标结合时,KDA_disc和KDA_cont对多种专家标注的MCQ质量度量指标具有较强的预测能力。

英文摘要:

The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value. They fail to evaluate the MCQ's ability to assess the student's knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ's answerability given knowledge of the target fact. Specifically, we first show how to measure KDA based on student responses from a human survey. Then, we propose two automatic evaluation metrics, KDA_disc and KDA_cont, that approximate KDA by leveraging pre-trained language models to imitate students' problem-solving behavior. Through our human studies, we show that KDA_disc and KDA_soft have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts. Furthermore, when combined with n-gram based similarity metrics, KDA_disc and KDA_cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑