Deep neural networks may easily memorize noisy labels present in real-world data, which degrades their ability to generalize. It is therefore important to track and evaluate the robustness of models against noisy label memorization. We propose a metric, called susceptibility, to gauge such memorization for neural networks. Susceptibility is simple and easy to compute during training. Moreover, it does not require access to ground-truth labels and it only uses unlabeled data. We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric. Finally, we show through extensive experiments on datasets with synthetic and real-world label noise that one can utilize susceptibility and the overall training accuracy to distinguish models that maintain a low memorization on the training set and generalize well to unseen clean data.
Revisiting Sample Selection Approach to Positive-Unlabeled Learning: Turning Unlabeled Data into Positive rather than Negative
RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能中心）；Center for Artificial Intelligence, University of Technology Sydney（悉尼科技大学人工智能中心）；University of Tokyo（东京大学）
A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective
Department of Computer and Data Sciences（计算机与数据科学系）；Case Western Reserve University（凯斯西储大学）；Department of Computer Science & Engineering（计算机科学与工程系）；Texas A&M University（德克萨斯大学）；Department of Biochemistry（生物化学系）；Center for RNA Science and Therapeutics（RNA科学与治疗中心）；Department of Biomedical Engineering（生物医学工程系）
CrossSplit: Mitigating Label Noise Memorization through Data Splitting
Samsung Advanced Institute of Technology (SAIT)（三星高级技术研究院（SAIT））；SAIT AI Lab（三星高级技术研究院人工智能实验室）；Mila（米拉研究所）；Université de Montreal（蒙特利尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能讲席）
On the Dynamics & Transferability of Latent Generalization during Memorization
Centre for Artificial Intelligence Research, Department of Computer Science & Information Systems, Birla Institute of Technology & Science Pilani（人工智能研究中心，计算机科学与信息系统系，比拉理工学院）
Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Elmore Family School of Electrical and Computer Engineering, Purdue University（普渡大学埃尔默电气与计算机工程学院）
Learning to Generate Gradients for Test-Time Adaptation via Test-Time Training Layers
South China University of Technology（华南理工大学）；Nanyang Technological University（南洋理工大学）；Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳北理莫斯科大学人工智能研究院）
Training Production Language Models without Memorizing User Data
Google LLC（谷歌有限责任公司）
本文提出了第一个消费者规模的下一个词预测（NWP）模型，该模型使用联邦学习（FL）进行训练，同时利用差分私有联邦平均（DP-FedAvg）技术。已经有关于构建实用FL基础设施的先前工作，包括展示使用这种基础设施在移动设备上训练语言模型的可行性的工作。它也被证明（在公共语料库上的模拟），它是可能的训练NWP模型与用户级差分隐私使用DP-FedAvg算法。然而，在现实世界的生产环境中，在一个异构的手机上，用DP-FedAvg训练生产质量的NWP模型需要解决许多挑战。例如，协调中央服务器必须在每轮开始时跟踪可用的设备，并从它们中随机均匀地采样设备，同时确保\emph{secrecy of the sample}等。与我们所知道的所有先前的以隐私为中心的FL工作不同，我们首次展示了用于在FL中训练生产神经网络的差异隐私机制的部署，以及生产培训基础设施的仪器化，以执行对非故意记忆的端到端经验测量。
A Multi-layered Acoustic Tokenizing Deep Neural Network (MAT-DNN) for Unsupervised Discovery of Linguistic Units and Generation of High Quality Features
Online Continual Adaptation with Active Self-Training
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）
