arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.04386cs.LG
收藏

Machine Learning Capability:使用案例难度的标准化度量,用于监督机器学习的个性化部署

Machine Learning Capability: A standardized metric using case difficulty with applications to individualized deployment of supervised machine learning

University of Calgary(卡尔加里大学);Cumming School of Medicine(卡明医学院)

Adrienne Kline, Joon Lee

摘要:

模型评估是有监督机器学习分类分析中的一个关键组成部分。传统的度量标准目前不包括病例难度。这使得分类结果无法进行基准化。项目反应理论(IRT)和计算机自适应测试(CAT)与机器学习可以独立于最终分类结果的基准数据集。这提供了关于评估效用的高水平的案例级信息。为了展示,使用了两个数据集:1)健康相关和2)物理科学。对于健康数据集,使用双参数IRT模型,对于物理科学数据集,使用多变量IRT模型来分析预测特征,并将每个病例置于难度连续体上。CAT的方法被用来确定算法的性能和适用性的新数据。这种方法提供了一种有效的基准数据方法,只使用一小部分数据集(不到1%),计算效率比传统指标高22- 60倍。这种称为机器学习能力(MLC)的新指标具有额外的好处,因为它对结果分类没有偏见,并且是在数据集内和跨数据集进行模型比较的标准化方式。MLC提供了关于监督机器学习算法的限制的度量。在算法不足的情况下,决策需要其他输入。

英文摘要:

Model evaluation is a critical component in supervised machine learning classification analyses. Traditional metrics do not currently incorporate case difficulty. This renders the classification results unbenchmarked for generalization. Item Response Theory (IRT) and Computer Adaptive Testing (CAT) with machine learning can benchmark datasets independent of the end-classification results. This provides high levels of case-level information regarding evaluation utility. To showcase, two datasets were used: 1) health-related and 2) physical science. For the health dataset a two-parameter IRT model, and for the physical science dataset a polytonomous IRT model, was used to analyze predictive features and place each case on a difficulty continuum. A CAT approach was used to ascertain the algorithms' performance and applicability to new data. This method provides an efficient way to benchmark data, using only a fraction of the dataset (less than 1%) and 22-60x more computationally efficient than traditional metrics. This novel metric, termed Machine Learning Capability (MLC) has additional benefits as it is unbiased to outcome classification and a standardized way to make model comparisons within and across datasets. MLC provides a metric on the limitation of supervised machine learning algorithms. In situations where the algorithm falls short, other input(s) are required for decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑