LLM在教育评估中是否给出心理测量学上合理的回答?
Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?
AI总结:
本文基于经典测试理论与项目反应理论,评估18个指令微调LLM在三科选择题上的心理测量学合理性,发现温度缩放可改善分布,但零样本相关性仍不足以替代人类预试。
摘要:
了解应试者如何回答教育评估中的题目,对于试卷开发、评估题目质量以及提升测试效度至关重要。然而,这一过程通常需要开展涉及人类参与者的大规模预试研究。如果大型语言模型(LLM)对测试题目表现出类似人类的作答行为,就有可能将其用作预试参与者,从而加快试卷开发。本文中,我们使用两个公开的选择题数据集,涵盖阅读、美国历史和经济学三个学科,评估了18个指令微调LLM回答的类人性,即心理测量学合理性。我们的方法建立在教育评估中常用的两个心理测量学理论框架之上:经典测试理论和项目反应理论。结果表明,尽管较大的模型过度自信,但在使用温度缩放进行校准后,其回答分布可以更接近人类。此外,我们发现,与其他学科相比,LLM在阅读理解题目上与人类的相关性更好。不过,总体而言这些相关性并不很强,表明不应在零样本设置下使用LLM进行教育评估预试。
英文摘要:
Knowing how test takers answer items in educational assessments is essential for test development, to evaluate item quality, and to improve test validity. However, this process usually requires extensive pilot studies with human participants. If large language models (LLMs) exhibit human-like response behavior to test items, this could open up the possibility of using them as pilot participants to accelerate test development. In this paper, we evaluate the human-likeness or psychometric plausibility of responses from 18 instruction-tuned LLMs with two publicly available datasets of multiple-choice test items across three subjects: reading, U.S. history, and economics. Our methodology builds on two theoretical frameworks from psychometrics which are commonly used in educational assessment, classical test theory and item response theory. The results show that while larger models are excessively confident, their response distributions can be more human-like when calibrated with temperature scaling. In addition, we find that LLMs tend to correlate better with humans in reading comprehension items compared to other subjects. However, the correlations are not very strong overall, indicating that LLMs should not be used for piloting educational assessments in a zero-shot setting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。