arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.01131cs.LG
收藏

概率分类中的预测多重性

Predictive Multiplicity in Probabilistic Classification

Harvard University(哈佛大学);DeepMind(深度思维公司);U.C. San Diego(加利福尼亚大学圣迭戈分校)

Jamelle Watson-Daniels, David C. Parkes, Berk Ustun

AI总结:

本文提出一个测量概率分类中预测多重性的框架,引入捕获竞争模型间风险估计变化的度量及基于优化的计算方法,并分析了其与现实数据特征的关系,强调需更广泛报告预测多重性。

摘要:

机器学习模型常被用于为现实世界风险评估任务提供信息:预测消费者违约风险,预测一个人是否患有严重疾病,或预测一个人出庭的风险。给定在预测任务中表现几乎同样好的多个模型,这些模型间的预测在多大程度上存在差异?如果相似模型的预测相对一致,那么选择优化惩罚损失的模型这一标准方法就足够了。但如果相似模型的预测差异显著呢?在机器学习中,这被称为预测多重性,即由近似最优的竞争模型所分配的冲突预测的普遍性。在本文中,我们提出了一个用于测量概率分类(预测正结果的概率)中预测多重性的框架。我们引入了捕获竞争模型集合上风险估计变化的度量,并开发了基于优化的方法,以高效可靠地计算凸经验风险最小化问题的这些度量。我们展示了现实任务中预测多重性的发生率和普遍性。此外,我们通过分析预测多重性与数据集特征(离群点、可分性以及多数-少数结构)之间的关系,深入探讨了预测多重性是如何产生的。我们的结果强调需要更广泛地报告预测多重性。

英文摘要:

Machine learning models are often used to inform real world risk assessment tasks: predicting consumer default risk, predicting whether a person suffers from a serious illness, or predicting a person's risk to appear in court. Given multiple models that perform almost equally well for a prediction task, to what extent do predictions vary across these models? If predictions are relatively consistent for similar models, then the standard approach of choosing the model that optimizes a penalized loss suffices. But what if predictions vary significantly for similar models? In machine learning, this is referred to as predictive multiplicity i.e. the prevalence of conflicting predictions assigned by near-optimal competing models. In this paper, we present a framework for measuring predictive multiplicity in probabilistic classification (predicting the probability of a positive outcome). We introduce measures that capture the variation in risk estimates over the set of competing models, and develop optimization-based methods to compute these measures efficiently and reliably for convex empirical risk minimization problems. We demonstrate the incidence and prevalence of predictive multiplicity in real-world tasks. Further, we provide insight into how predictive multiplicity arises by analyzing the relationship between predictive multiplicity and data set characteristics (outliers, separability, and majority-minority structure). Our results emphasize the need to report predictive multiplicity more widely.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑