重新评估如何比较和改进机器学习模型的校准
Reassessing How to Compare and Improve the Calibration of Machine Learning Models
AI总结:
本研究重新评估机器学习模型校准指标的报告方式,提出基于Bregman散度分解的可靠性图扩展以联合可视化校准与泛化误差,并证明相关理论结果。
摘要:
如果机器学习模型对某个结果的预测概率与该结果在模型预测条件下的观测频率相匹配,则该模型是校准的。随着机器学习模型的影响持续扩展到各个领域,这一性质变得越来越重要。因此,目前有大量近期论文致力于测量和改进(特别是深度学习)模型的校准。在这项工作中,我们重新评估了近期文献中校准指标的报告方式。我们表明,除非校准和预测指标(即测试准确率)伴随额外的泛化指标(如负对数似然),否则存在看似最先进的平凡重校准方法。然后,我们利用基于校准的Bregman散度分解,开发了可靠性图的新扩展,该扩展联合可视化校准和泛化误差,并展示了我们的可视化如何用于检测校准与泛化之间的权衡。在此过程中,我们证明了关于Bregman散度的完全校准误差与置信度校准误差之间关系的新结果。我们还建立了用于我们可视化方法中校准误差的核回归估计量的一致性,该结果推广了文献中现有的一致性结果。
英文摘要:
A machine learning model is calibrated if its predicted probability for an outcome matches the observed frequency for that outcome conditional on the model prediction. This property has become increasingly important as the impact of machine learning models has continued to spread to various domains. As a result, there are now a dizzying number of recent papers on measuring and improving the calibration of (specifically deep learning) models. In this work, we reassess the reporting of calibration metrics in the recent literature. We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-art unless calibration and prediction metrics (i.e. test accuracy) are accompanied by additional generalization metrics such as negative log-likelihood. We then use a calibration-based decomposition of Bregman divergences to develop a new extension to reliability diagrams that jointly visualizes calibration and generalization error, and show how our visualization can be used to detect trade-offs between calibration and generalization. Along the way, we prove novel results regarding the relationship between full calibration error and confidence calibration error for Bregman divergences. We also establish the consistency of the kernel regression estimator for calibration error used in our visualization approach, which generalizes existing consistency results in the literature.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。