多类校准评估与概率预测的线性对数几率校准函数 recalibration
Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Function
AI总结：
本文提出多类线性对数几率校准方法，通过似然比假设检验和无需模型内部访问实现对单个模型校准的评估，适用于广泛分类问题并易于解释。
摘要：
机器生成的概率预测在现代分类任务如图像分类中至关重要。当模型的预测概率与观察到的事件频率相对应时，模型是良好校准的。尽管多类校准方法的需求存在，但现有方法存在三个限制：（i）只能比较两个或多个模型之间的校准，而不能直接评估单个模型的校准；（ii）需要模型内部的访问权限，例如访问神经网络层内的logit尺度预测；（iii）提供的输出对人类分析师来说难以理解。为克服这些限制，我们提出了多类线性对数几率（MCLLO）校准方法，该方法（i）包含似然比假设检验来评估校准，（ii）不需要模型内部的访问权限，因此适用于广泛分类问题，（iii）可以轻松解释。我们通过模拟和三个涉及卷积神经网络图像分类、随机森林肥胖分析和回归建模生态的现实案例研究，展示了MCLLO方法的有效性。我们通过使用我们的假设检验和现有的校准度量预期校准误差，将MCLLO与四种比较校准技术进行比较，以显示我们的方法单独使用和与其他方法结合使用时都表现良好。
英文摘要：
Machine-generated probability predictions are essential in modern classification tasks such as image classification. A model is well calibrated when its predicted probabilities correspond to observed event frequencies. Despite the need for multicategory recalibration methods, existing methods are limited to (i) comparing calibration between two or more models rather than directly assessing the calibration of a single model, (ii) requiring under-the-hood model access, e.g., accessing logit-scale predictions within the layers of a neural network, and (iii) providing output which is difficult for human analysts to understand. To overcome (i)-(iii), we propose Multicategory Linear Log Odds (MCLLO) recalibration, which (i) includes a likelihood ratio hypothesis test to assess calibration, (ii) does not require under-the-hood access to models and is thus applicable on a wide range of classification problems, and (iii) can be easily interpreted. We demonstrate the effectiveness of the MCLLO method through simulations and three real-world case studies involving image classification via convolutional neural network, obesity analysis via random forest, and ecology via regression modeling. We compare MCLLO to four comparator recalibration techniques utilizing both our hypothesis test and the existing calibration metric Expected Calibration Error to show that our method works well alone and in concert with other methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
