arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.06245cs.LG
收藏

使用其前驱网络校准深度神经网络

Calibrating a Deep Neural Network with Its Predecessors

University of Sydney(悉尼大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织数据61研究所)

Linwei Tao, Minjing Dong, Daochang Liu, Changming Sun, Chang Xu

AI总结:

针对早停法无法有效校准网络的问题,提出前驱组合搜索(PCS)正则化方法,通过搜索早期训练阶段的模块前驱组合来改善校准,在多数据集上实现SOTA性能并提升模型鲁棒性。

摘要:

置信度校准——即校准神经网络输出概率分布的过程——对于此类网络的安全关键应用至关重要。近期研究验证了误校准与过拟合之间的联系。然而,作为缓解过拟合的著名技术,早停法却无法有效校准网络。在本研究中,我们探讨了早停法的局限性,并综合考虑每个独立模块全面分析了网络的过拟合问题。随后,我们提出了一种新颖的正则化方法——前驱组合搜索(PCS),通过搜索最佳拟合模块前驱的组合来改善校准,其中模块前驱是指具有来自早期训练阶段权重参数的对应网络模块。PCS在多个数据集和架构上实现了最先进的校准性能。此外,PCS还提升了模型在数据集分布偏移下的鲁棒性。

英文摘要:

Confidence calibration - the process to calibrate the output probability distribution of neural networks - is essential for safety-critical applications of such networks. Recent works verify the link between mis-calibration and overfitting. However, early stopping, as a well-known technique to mitigate overfitting, fails to calibrate networks. In this work, we study the limitions of early stopping and comprehensively analyze the overfitting problem of a network considering each individual block. We then propose a novel regularization method, predecessor combination search (PCS), to improve calibration by searching a combination of best-fitting block predecessors, where block predecessors are the corresponding network blocks with weight parameters from earlier training stages. PCS achieves the state-of-the-art calibration performance on multiple datasets and architectures. In addition, PCS improves model robustness under dataset distribution shift.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑