arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.08823cs.LGstat.AP
收藏

AutoIRT:利用自动化机器学习校准项目反应理论模型

AutoIRT: Calibrating Item Response Theory Models with Automated Machine Learning

Duolingo(多邻国)

James Sharpnack, Phoebe Mulcaire, Klinton Bicknell, Geoff LaFlair, Kevin Yancey

AI总结:

本研究提出可兼容开箱即用AutoML工具的AutoIRT多阶段拟合流程,基于MCEM外循环和双阶段内循环校准IRT模型,在Duolingo English Test上实现了优于现有方法的校准效果、预测性能与评分准确性。

摘要:

项目反应理论(IRT)是一类可解释的因子模型,广泛应用于计算机化自适应测试(CATs),例如语言能力测试。传统上,这类模型通过参数化混合效应模型拟合,以预测考生答对测试项目(即题目)的概率。这类模型的神经网络扩展版本(如BertIRT)需要专门的架构与参数调优。我们提出了一种多阶段拟合流程,可与开箱即用的自动化机器学习(AutoML)工具兼容。该流程基于蒙特卡洛期望最大化(MCEM)外循环,搭配双阶段内循环:先利用题目特征训练非参数化AutoML评分模型,随后训练题目特定的参数化模型。这极大加快了测试评分的建模工作流。我们将其应用于高利害性在线英语能力测试Duolingo English Test,验证了其有效性。结果表明,与现有方法(非解释性IRT模型及BERT-IRT等解释性IRT模型)相比,所提模型通常校准效果更好、预测性能更优,评分也更准确。研究过程中,我们还对用于CATs题目参数校准的机器学习方法进行了简要综述。

英文摘要:

Item response theory (IRT) is a class of interpretable factor models that are widely used in computerized adaptive tests (CATs), such as language proficiency tests. Traditionally, these are fit using parametric mixed effects models on the probability of a test taker getting the correct answer to a test item (i.e., question). Neural net extensions of these models, such as BertIRT, require specialized architectures and parameter tuning. We propose a multistage fitting procedure that is compatible with out-of-the-box Automated Machine Learning (AutoML) tools. It is based on a Monte Carlo EM (MCEM) outer loop with a two stage inner loop, which trains a non-parametric AutoML grade model using item features followed by an item specific parametric model. This greatly accelerates the modeling workflow for scoring tests. We demonstrate its effectiveness by applying it to the Duolingo English Test, a high stakes, online English proficiency test. We show that the resulting model is typically more well calibrated, gets better predictive performance, and more accurate scores than existing methods (non-explanatory IRT models and explanatory IRT models like BERT-IRT). Along the way, we provide a brief survey of machine learning methods for calibration of item parameters for CATs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑