发表机构
University of Münster（明斯特大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出用随机标签预测头研究深度神经网络分类任务中的记忆，通过其性能估计拉德马赫复杂度，分析泛化与过拟合，还基于此提出正则化技术，挑战了过拟合等同于记忆的传统假设，揭示减少记忆对泛化的复杂影响。
AI 中文摘要
我们引入了一种直接且有效的方法来实证研究深度神经网络在分类任务中的记忆情况。我们的方法是用辅助随机标签增强每个训练样本，然后由随机标签预测头（RLP-head）进行预测。RLP-head可附加在网络的任意深度，从相应中间表示预测随机标签，从而分析记忆能力如何跨层演变。通过将RLP-head性能解释为拉德马赫复杂度的实证估计，我们得到了样本级记忆和模型容量的直接度量。利用此随机标签准确率指标分析不同模型和数据集的泛化与过拟合情况。在此基础上，我们进一步提出基于RLP-head输出的新型正则化技术，可显著减少记忆。有趣的是，实验表明减少记忆对泛化的影响取决于数据集和训练设置。这些发现挑战了过拟合等同于记忆的传统假设，并提出了调和这些看似矛盾结果的新假设。
英文摘要
We introduce a straightforward yet effective method to empirically study memorization in deep neural networks for classification tasks. Our approach augments each training sample with auxiliary random labels, which are then predicted by a random label prediction head (RLP-head). RLP-heads can be attached at arbitrary depths of a network, predicting random labels from the corresponding intermediate representation and thereby enabling analysis of how memorization capacity evolves across layers. By interpreting the RLP-head performance as an empirical estimate of Rademacher complexity, we obtain a direct measure of both sample-level memorization and model capacity. We leverage this random label accuracy metric to analyze generalization and overfitting in different models and datasets. Building on this approach, we further propose a novel regularization technique based on the output of the RLP-head, which demonstrably reduces memorization. Interestingly, our experiments reveal that reducing memorization can either improve or impair generalization, depending on the dataset and training setup. These findings challenge the traditional assumption that overfitting is equivalent to memorization and suggest new hypotheses to reconcile these seemingly contradictory results. The source code is available at https://github.com/MarlonBecker/RandomLabelHeads
Journal refICLR 2026
