JUCAL: 在分类任务中联合校准偶然不确定性和本质不确定性
JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks
AI总结：
JUCAL通过联合校准偶然和本质不确定性，提升分类任务中预测不确定性的准确性。
摘要：
我们研究了训练好的分类器集合的后校准不确定性。具体来说，我们考虑了偶然不确定性（标签噪声）和本质不确定性（模型）两种不确定性。在分类中，最流行且广泛使用的校准方法包括温度缩放（即池后校准）和符合方法。然而，这些校准方法的主要缺点是它们不能平衡偶然和本质不确定性的比例。不平衡这些不确定性可能导致预测不确定性严重失真，导致某些输入区域过于自信的预测，而在其他区域则过于不自信。为了解决这一缺点，我们提出了一种简单但强大的校准算法，即联合不确定性校准（JUCAL），该算法联合校准偶然和本质不确定性。JUCAL通过在验证/校准数据集上优化负对数似然（NLL）来联合校准两个常数，以加权和缩放本质和偶然不确定性。JUCAL可以应用于任何训练好的分类器集合（例如，transformers、CNNs或基于树的方法），计算开销极小，且不需要访问模型的内部参数。我们实验性地评估了JUCAL在各种文本分类任务上的表现，针对不同大小的集合和不同的融合策略。我们的实验表明，JUCAL在所有考虑的分类任务中均显著优于现有最佳校准方法，将NLL和预测集大小分别减少高达15%和20%。有趣的是，即使将JUCAL应用于大小为5的集合，也能在NLL和预测集大小方面超越温度缩放的大小达50的集合，从而在推理成本上减少高达10倍。因此，我们提出JUCAL作为分类中校准集合的新方法。
英文摘要：
We study post-calibration uncertainty for trained ensembles of classifiers. Specifically, we consider both aleatoric (label noise) and epistemic (model) uncertainty. Among the most popular and widely used calibration methods in classification are temperature scaling (i.e., pool-then-calibrate) and conformal methods. However, the main shortcoming of these calibration methods is that they do not balance the proportion of aleatoric and epistemic uncertainty. Not balancing these uncertainties can severely misrepresent predictive uncertainty, leading to overconfident predictions in some input regions while being underconfident in others. To address this shortcoming, we present a simple but powerful calibration algorithm Joint Uncertainty Calibration (JUCAL) that jointly calibrates aleatoric and epistemic uncertainty. JUCAL jointly calibrates two constants to weight and scale epistemic and aleatoric uncertainties by optimizing the negative log-likelihood (NLL) on the validation/calibration dataset. JUCAL can be applied to any trained ensemble of classifiers (e.g., transformers, CNNs, or tree-based methods), with minimal computational overhead, without requiring access to the models' internal parameters. We experimentally evaluate JUCAL on various text classification tasks, for ensembles of varying sizes and with different ensembling strategies. Our experiments show that JUCAL significantly outperforms SOTA calibration methods across all considered classification tasks, reducing NLL and predictive set size by up to 15% and 20%, respectively. Interestingly, even applying JUCAL to an ensemble of size 5 can outperform temperature-scaled ensembles of size up to 50 in terms of NLL and predictive set size, resulting in up to 10 times smaller inference costs. Thus, we propose JUCAL as a new go-to method for calibrating ensembles in classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
