arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.05327cs.CL
收藏

模型不确定性能否作为多项选择题项目难度的代理?

Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?

University of Groningen(格罗宁根大学)

Leonidas Zotos, Hedderik van Rijn, Malvina Nissim

AI总结:

本研究探索生成式大模型的不确定性度量与多项选择题实际学生作答分布之间的相关性,发现相关性微弱且因问题类型和答案正误而异,为利用模型不确定性作为题目难度代理提供了新途径。

摘要:

估计多项选择题的难度对必须花费大量时间创建和测试试题的教育工作者,以及希望进行练习的学习者来说,将是极大的帮助。迄今为止,监督式难度估计方法产生了褒贬不一的结果。在本研究中,我们利用生成式大模型在回答问题时可能被视为弱点的一个方面,即其不确定性,并借此探索两种不同的不确定性度量与实际学生作答分布之间的相关性。虽然我们观察到存在一些微弱的相关性,但也发现模型在回答正确与错误时的行为有所不同,并且根据包含在我们包含451道题目的生物心理学课程的细粒度、此前未使用过的数据集中的不同问题类型,相关性存在显著差异。在讨论我们的发现时,我们还建议了进一步利用模型不确定性作为项目难度附加代理的潜在途径。

英文摘要:

Estimating the difficulty of multiple-choice questions would be great help for educators who must spend substantial time creating and piloting stimuli for their tests, and for learners who want to practice. Supervised approaches to difficulty estimation have yielded to date mixed results. In this contribution we leverage an aspect of generative large models which might be seen as a weakness when answering questions, namely their uncertainty, and exploit it towards exploring correlations between two different metrics of uncertainty, and the actual student response distribution. While we observe some present but weak correlations, we also discover that the models' behaviour is different in the case of correct vs wrong answers, and that correlations differ substantially according to the different question types which are included in our fine-grained, previously unused dataset of 451 questions from a Biopsychology course. In discussing our findings, we also suggest potential avenues to further leverage model uncertainty as an additional proxy for item difficulty.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑