arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.08232cs.CLcs.AI
收藏

LLM 能否可靠地模拟真实学生在数学与阅读理解方面的能力?

Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?

Mohamed bin Zayed University of Artificial Intelligence(莫莫德·本·扎耶德人工智能大学)

KV Aditya Srivatsa, Kaushal Kumar Maurya, Ekaterina Kochmar

AI总结:

研究通过 NAEP 数据集和 IRT 模型评估 11 个 LLM 作为学生代理的可靠性,发现无指导时强模型优于普通学生,使用年级强制提示后仍无模型-提示对能跨学科年级匹配真实学生,呼吁新训练评估策略。

摘要:

大型语言模型(LLMs)在智能辅导系统(ITSs)的开发和测试题的试测中,正越来越多地被用作代理学生。然而,这些代理学生能在多大程度上准确模拟真实学生的行为和特征,仍是一个悬而未决的问题。为了调查这一点,我们收集了来自国家教育进展评估(NAEP)的 489 个题目的数据集,涵盖 4、8 和 12 年级的数学与阅读理解。然后,我们应用项目反应理论(IRT)模型,将 11 个多样化且最先进的 LLMs 定位在与真实学生群体相同的能力量表上。我们的发现表明,在没有指导的情况下,强大的通用模型在每个年级的表现始终优于平均水平的学生,而较弱或领域不匹配的模型可能会偶然对齐。使用年级强制提示改变了模型的性能,但它们是否与平均年级水平的学生对齐仍然高度依赖于模型和提示:没有评估的模型-提示对在所有学科和年级中都能符合要求,这凸显了新训练和评估策略的需求。最后,我们基于发现提供了选择可行代理的指南。

英文摘要:

Large Language Models (LLMs) are increasingly used as proxy students in the development of Intelligent Tutoring Systems (ITSs) and in piloting test questions. However, to what extent these proxy students accurately emulate the behavior and characteristics of real students remains an open question. To investigate this, we collected a dataset of 489 items from the National Assessment of Educational Progress (NAEP), covering mathematics and reading comprehension in grades 4, 8, and 12. We then apply an Item Response Theory (IRT) model to position 11 diverse and state-of-the-art LLMs on the same ability scale as real student populations. Our findings reveal that, without guidance, strong general-purpose models consistently outperform the average student at every grade, while weaker or domain-mismatched models may align incidentally. Using grade-enforcement prompts changes models' performance, but whether they align with the average grade-level student remains highly model- and prompt-specific: no evaluated model-prompt pair fits the bill across subjects and grades, underscoring the need for new training and evaluation strategies. We conclude by providing guidelines for the selection of viable proxies based on our findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑