EnsLoss:用于防止分类中过拟合的随机校准损失集成方法
EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification
AI总结:
本文提出EnsLoss方法,将集成学习扩展到损失函数层面,通过随机校准损失导数在单次训练中实现损失集成,并在多个数据集上验证了其优于固定损失方法的性能。
摘要:
采用计算上可行的替代损失进行经验风险最小化(ERM)是一种被广泛接受的分类方法。值得注意的是,损失函数的凸性与校准性(CC)性质能够确保ERM在最大化准确率方面具有一致性,从而为替代损失提供了广泛的选择空间。本文提出了一种新颖的集成方法,名为EnsLoss,它将集成学习的概念扩展到ERM框架内的损失函数组合中。我们方法的一个关键特征是考虑保持组合损失的“合法性”,即确保其CC性质。具体而言,我们首先将损失的CC条件转化为损失导数形式,从而绕过对显式损失函数的需求,直接生成经过校准的损失导数。因此,受Dropout启发,EnsLoss通过一次训练过程即可实现损失集成,该过程采用双重随机梯度下降(即随机批次样本和随机校准损失导数)。我们从理论上建立了该方法的统计一致性,并对其优势进行了深入分析。通过在14个OpenML表格数据集和46个图像数据集上使用多种深度学习架构进行实验,证明了EnsLoss相较于固定损失方法在数值上的有效性。Python代码库和源代码已在GitHub上公开,地址为https://github.com/statmlben/ensloss。
英文摘要:
Empirical risk minimization (ERM) with a computationally feasible surrogate loss is a widely accepted approach for classification. Notably, the convexity and calibration (CC) properties of a loss function ensure consistency of ERM in maximizing accuracy, thereby offering a wide range of options for surrogate losses. In this article, we propose a novel ensemble method, namely EnsLoss, which extends the ensemble learning concept to combine loss functions within the ERM framework. A key feature of our method is the consideration on preserving the "legitimacy" of the combined losses, i.e., ensuring the CC properties. Specifically, we first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions and directly generating calibrated loss-derivatives. Therefore, inspired by Dropout, EnsLoss enables loss ensembles through one training process with doubly stochastic gradient descent (i.e., random batch samples and random calibrated loss-derivatives). We theoretically establish the statistical consistency of our approach and provide insights into its benefits. The numerical effectiveness of EnsLoss compared to fixed loss methods is demonstrated through experiments on a broad range of 14 OpenML tabular datasets and 46 image datasets with various deep learning architectures. Python repository and source code are available on GitHub at https://github.com/statmlben/ensloss.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。