Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $β^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curve (ICC). In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $β^3$-IRT to assess the ability of machine learning classifiers. This novel application results in a new metric for evaluating the quality of the classifier's probability estimates, based on the inferred difficulty and discrimination of data instances.
IRTCI: Item Response Theory for Categorical Imputation
Northwestern University(西北大学)
大多数数据集都存在部分或完整的缺失值,这对测试数据的可用模型以及可以从数据中进行的任何统计推断具有下游限制。已设计了几种插补技术,用替代值替换缺失数据。各种方法对计算临床评分、模型构建和模型检验具有影响。这里展示的工作提供了一种基于项目反应理论(IRT)的分类插补的新方法,并将其与机器学习领域目前使用的几种方法进行了比较,包括k-最近邻(kNN),多重插补链式方程(MICE)和Amazon Web Services(AWS)深度学习方法Datawig。比较这些技术的分析进行了三个不同的数据集,代表序数,名义和二进制类别。对数据进行了修改,使其在数据缺失比例和缺失数据的系统化方面也有所不同。进行了两种不同的性能评估:重现缺失值的准确性和使用插补数据的预测性能。结果表明,新的方法,项目反应理论分类插补(IRTCI),表现相当不错,目前使用的方法相比,在许多情况下,优于他们中的几个。鉴于新方法的理论基础,以及用于确定缺失细胞类别归属的概率项的独特生成,IRTCI为当前方法提供了一种可行的替代方案。
Faculdade de Computação, Universidade Federal do Pará(帕拉联邦大学计算机学院);Centro de Informática, Universidade Federal de Pernambuco(伯南布哥联邦大学信息中心);Instituto Tecnológico Vale(淡水河谷技术研究所)