发表机构
Southeast University; Saint Francis University; City University of Hong Kong（东南大学; 圣弗朗西斯大学; 香港城市大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对深度神经网络训练中易过度自信的问题，提出EUA-Cal方法，利用早期模型作为不确定性锚点，通过两类正则化缓解过度自信，在8种模型的两类任务上性能优于现有校准方法。
AI 中文摘要
深度神经网络（包括大语言模型）在各类任务中已取得显著性能，但在训练或微调过程中易出现过度自信问题。本研究观察到不同模型均存在一致现象：早期模型校准效果更好，后续训练或微调仅能带来微小的准确率提升，却会大幅增加校准误差。分析表明，早期模型在预测和特征层面保留了不确定性感知能力，该能力会随持续训练逐渐丧失。为避免训练消除这种不确定性感知，我们提出EUA-Cal这一新方法，将早期模型用作校准的不确定性锚点。EUA-Cal引入早期预测正则化以保留早期预测不确定性，以及原型结构正则化以利用早期特征空间反映的不确定性，共同缓解过度自信问题。在图像分类和多项选择问答任务上对8种不同模型开展的大量实验表明，EUA-Cal的性能优于现有最先进的校准方法。
英文摘要
Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose \textbf{EUA-Cal}, a novel method that exploits the \textbf{E}arly model as an \textbf{U}ncertainty \textbf{A}nchor for \textbf{Cal}ibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.
Comments18 pages, 9 figures, 11 tables
