arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11206cs.LGcs.CRstat.ML
收藏

使用预测硬标签进行重训练可证明提升模型准确率

Retraining with Predicted Hard Labels Provably Increases Model Accuracy

Google Research(谷歌研究院);Google(谷歌);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Southern California(南加州大学)

Rudrajit Das, Inderjit S. Dhillon, Alessandro Epasto, Adel Javanmard, Jieming Mao, Vahab Mirrokni, Sujay Sanghavi, Peilin Zhong

AI总结:

本文首次从理论上证明在噪声标签下用模型自身预测硬标签重训练可提升准确率,并提出基于一致性的重训练方法,在不增加隐私代价下显著改善标签差分隐私训练效果。

摘要:

使用噪声标签训练的模型,其性能往往可以通过简单地用模型自身预测的硬标签(即1/0标签)进行重训练而得到改善。然而,目前对这一现象尚缺乏详细的理论刻画。在本文中,我们在带有随机损坏标签的线性可分二分类设定下对重训练进行了理论分析,并证明重训练能够提升最初使用给定(噪声)标签训练所获得的总体准确率。据我们所知,这是首个此类理论结果。重训练在改进具有本地标签差分隐私(DP)的训练中具有应用价值,而该训练过程涉及使用噪声标签。我们通过实验表明,仅对预测标签与给定标签一致的样本进行选择性重训练,能够在不增加额外隐私代价的情况下显著改善标签差分隐私训练;我们将此方法称为基于一致性的重训练。例如,在CIFAR-100上以ε=3的标签差分隐私训练ResNet-18时,采用基于一致性的重训练可获得超过6%的准确率提升。

英文摘要:

The performance of a model trained with noisy labels is often improved by simply \textit{retraining} the model with its \textit{own predicted hard labels} (i.e., 1/0 labels). Yet, a detailed theoretical characterization of this phenomenon is lacking. In this paper, we theoretically analyze retraining in a linearly separable binary classification setting with randomly corrupted labels given to us and prove that retraining can improve the population accuracy obtained by initially training with the given (noisy) labels. To the best of our knowledge, this is the first such theoretical result. Retraining finds application in improving training with local label differential privacy (DP) which involves training with noisy labels. We empirically show that retraining selectively on the samples for which the predicted label matches the given label significantly improves label DP training at no extra privacy cost; we call this consensus-based retraining. As an example, when training ResNet-18 on CIFAR-100 with $ε=3$ label DP, we obtain more than 6% improvement in accuracy with consensus-based retraining.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑