SMART：与项目反应理论对齐的模拟学生用于题目难度预测
SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction
AI总结：
本文提出SMART方法，利用直接偏好优化将模拟学生与指定能力对齐，通过LLM评分和IRT模型拟合预测开放式题目难度，解决了冷启动问题并在真实数据集上优于其他方法。
摘要：
项目（题目）难度在教育评估中起着至关重要的作用，能够准确高效地评估学生能力并进行个性化以最大化学习成果。传统上，估计项目难度成本高昂，需要真实学生对项目作答，随后拟合项目反应理论（IRT）模型以获取难度估计值。该方法也无法应用于先前未见项目的冷启动场景。在本研究中，我们提出了SMART（与IRT对齐的模拟学生），一种将模拟学生与指定能力对齐的新颖方法，随后可在模拟中用于预测开放式项目的难度。我们使用直接偏好优化（DPO）实现这种对齐，其中基于真实IRT模型下响应的可能性形成偏好对。我们通过生成数千个响应、使用基于大型语言模型（LLM）的评分模型评估它们，并将结果数据拟合到IRT模型中以获取项目难度估计值来进行模拟。通过在两个真实世界的学生响应数据集上进行广泛实验，我们表明SMART通过利用其改进的能力对齐，优于其他项目难度预测方法。
英文摘要：
Item (question) difficulties play a crucial role in educational assessments, enabling accurate and efficient assessment of student abilities and personalization to maximize learning outcomes. Traditionally, estimating item difficulties can be costly, requiring real students to respond to items, followed by fitting an item response theory (IRT) model to get difficulty estimates. This approach cannot be applied to the cold-start setting for previously unseen items either. In this work, we present SMART (Simulated Students Aligned with IRT), a novel method for aligning simulated students with instructed ability, which can then be used in simulations to predict the difficulty of open-ended items. We achieve this alignment using direct preference optimization (DPO), where we form preference pairs based on how likely responses are under a ground-truth IRT model. We perform a simulation by generating thousands of responses, evaluating them with a large language model (LLM)-based scoring model, and fit the resulting data to an IRT model to obtain item difficulty estimates. Through extensive experiments on two real-world student response datasets, we show that SMART outperforms other item difficulty prediction methods by leveraging its improved ability alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
