Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given.
In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).
Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?
Department of Mathematics, Imperial College London, UK；ML Lab, Capital Fund Management, France；Department of Physics, \'Ecole Polytechnique F\'ed\'erale de Lausanne (EPFL), Switzerland
On the Convergence Rate of Training Recurrent Neural Networks
Microsoft Research AI（微软研究院AI）；Stanford University（斯坦福大学）；Princeton University（普林斯顿大学）；UT-Austin（德克萨斯大学奥斯汀分校）；University of Washington（华盛顿大学）；Harvard University（哈佛大学）
Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Elmore Family School of Electrical and Computer Engineering, Purdue University（普渡大学埃尔默电气与计算机工程学院）
Reason to Rote: Rethinking Memorization in Reasoning
Department of ICS, Utrecht University（信息科学与计算系，乌特勒支大学）；MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Saarland Informatics Campus, Saarland University（萨尔兰州信息科学校区，萨尔兰州大学）
Discovering Causal Relations and Equations from Data
Universitat de València（瓦伦西亚大学）；German Aerospace Center（德国航空航天中心）；Technische Universität Berlin（柏林工业大学）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Bournemouth University（伯恩茅斯大学）；Heidelberg University（海德堡大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；New York University（纽约大学）
Systematic construction of continuous-time neural networks for linear dynamical systems
Institute of Advanced Study, Technical University of Munich（慕尼黑工业大学高等研究院）；Institute of Data-Science Foundations, Technische Universität Hamburg（汉堡工业大学数据科学基础研究所）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；Eindhoven University of Technology（埃因霍温理工大学）
