迈出一大步：去噪得分匹配中的大学习率防止记忆化

Early-Learning Regularization Prevents Memorization of Noisy Labels

We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classificatio…

Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference

Membership inference (MI) attacks exploit the fact that machine learning algorithms sometimes leak information about their training data through the learned model. In this work, we study membership inference in the white-box setting in order to exploit the internals of a model, which have not been effectively utilized by previous work. Leveraging new insights about how overfitting occurs in deep neural networks, we show how a model's idiosyncratic use of features can provide evidence for membership to white-box attackers---even when the model's black-box behavior appears to generalize well---a…

在惰性高维 regime 中训练的扩散模型的泛化、记忆与过拟合

Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime

本研究针对惰性高维 regime 中训练的扩散模型，推导了梯度流训练下的精确风险轨迹，揭示了其泛化、记忆与过拟合的阶段机制，为生成式建模提供了理论分析框架。

Robust and On-the-fly Dataset Denoising for Image Classification

Memorization in over-parameterized neural networks could severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are hard to avoid in extremely large datasets collected with weak supervision. We address this problem by reasoning counterfactually about the loss distribution of examples with uniform random labels had they were trained with the real examples, and use this information to remove noisy examples from the training set. First, we observe that examples with uniform random labels have higher losses when trained with stochastic gradient descent un…

学习随机标签预测作为神经网络复杂度度量

Learned Random Label Predictions as a Neural Network Complexity Metric

该研究通过引入可遗忘随机标签的多头CNN架构，将学习随机标签预测用作复杂度度量分析正则化效果、挑战传统CNN认知，并提出新型正则器减少样本记忆，为神经网络复杂度研究提供新方法。

扩散模型中记忆与泛化之间的可证明分离

Provable Separations between Memorization and Generalization in Diffusion Models

本文从统计估计和网络逼近两个视角证明扩散模型中记忆化与泛化之间的可证明分离，并据此提出基于剪枝的方法，在扩散Transformer中减少记忆化同时保持生成质量。