Does Learning Require Memorization? A Short Tale about a Long Tail
Apple（苹果公司）；Simons Institute for the Theory of Computing（西蒙斯计算理论研究所）；Google Research（谷歌研究院）
Vitaly Feldman
备注：
Significant revision: revised introduction/overview; added formal treatment of noise in the labels and explanation for the disparate effects of limiting memorization
State-of-the-art results on image recognition tasks are achieved using over-parameterized learning algorithms that (nearly) perfectly fit the training set and are known to fit well even random labels. This tendency to memorize the labels of the training data is not explained by existing theoretical analyses. Memorization of the training data also presents significant privacy risks when the training data contains sensitive personal information and thus it is important to understand whether such memorization is necessary for accurate learning.
We provide the first conceptual explanation and a theoretical model for this phenomenon. Specifically, we demonstrate that for natural data distributions memorization of labels is necessary for achieving close-to-optimal generalization error. Crucially, even labels of outliers and noisy labels need to be memorized. The model is motivated and supported by the results of several recent empirical works. In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed. Image and text data is known to be long-tailed and therefore our results establish a formal link between these empirical phenomena. Our results allow to quantify the cost of limiting memorization in learning and explain the disparate effects that privacy and model compression have on different subgroups.
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Meta FAIR；Center for Data Science, New York University（纽约大学数据科学中心）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Courant Institute, New York University（纽约大学柯朗研究所）
Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data
King Abdullah University of Science and Technology（卡塔尔国王 Abdullah 科学与技术大学）；Renmin University of China（中国人民大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Beijing Institute of Technology（北京理工大学）
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Mila – Quebec AI Institute（魁北克人工智能研究所）；University of Montreal（蒙特利尔大学）；Saarland University（萨尔大学）
Measures of Information Reflect Memorization Patterns
Delhi Technological University（德里理工大学）；Google Research India（谷歌研究院印度分部）；Amazon Web Services（亚马逊网络服务）；Technion – Israel Institute of Technology（以色列理工学院）；Carnegie Mellon University（卡内基梅隆大学）
A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective
Department of Computer and Data Sciences（计算机与数据科学系）；Case Western Reserve University（凯斯西储大学）；Department of Computer Science & Engineering（计算机科学与工程系）；Texas A&M University（德克萨斯大学）；Department of Biochemistry（生物化学系）；Center for RNA Science and Therapeutics（RNA科学与治疗中心）；Department of Biomedical Engineering（生物医学工程系）
Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?
Department of Mathematics, Imperial College London, UK；ML Lab, Capital Fund Management, France；Department of Physics, \'Ecole Polytechnique F\'ed\'erale de Lausanne (EPFL), Switzerland
