Over-parameterized deep neural networks are able to achieve excellent training accuracy while maintaining a small generalization error. It has also been found that they are able to fit arbitrary labels, and this behaviour is referred to as the phenomenon of memorization. In this work, we study the phenomenon of memorization with turn-over dropout, an efficient method to estimate influence and memorization, for data with true labels (real data) and data with random labels (random data). Our main findings are: (i) For both real data and random data, the optimization of easy examples (e.g., real data) and difficult examples (e.g., random data) are conducted by the network simultaneously, with easy ones at a higher speed; (ii) For real data, a correct difficult example in the training dataset is more informative than an easy one. By showing the existence of memorization on random data and real data, we highlight the consistency between them regarding optimization and we emphasize the implication of memorization during optimization.
Adynamical systems view of training generativemodels and the memorization phenomenon
International Institute for Theoretical Sciences（理论科学国际研究所）；Department of Computer Science and Automation（计算机科学与自动化系）；Indian Institute of Science（印度科学研究所）；Department of Electrical Engineering（电气工程系）；Indian Institute of Technology Bombay（博亚理工大学）
Understanding Private Learning From Feature Perspective
Department of Computer Science and Engineering, State University of New York at Buffalo（纽约州立大学布法罗分校计算机科学与工程系）；Division of CEMSE, King Abdullah University of Science and Technology（卡普兰大学科学与技术大学CEMSE分校）；Institute of Artificial Intelligence and Blockchain, Guangzhou University（广州大学人工智能与区块链研究院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective
Department of Computer and Data Sciences（计算机与数据科学系）；Case Western Reserve University（凯斯西储大学）；Department of Computer Science & Engineering（计算机科学与工程系）；Texas A&M University（德克萨斯大学）；Department of Biochemistry（生物化学系）；Center for RNA Science and Therapeutics（RNA科学与治疗中心）；Department of Biomedical Engineering（生物医学工程系）
Memorization Control in Diffusion Models from Denoising-centric Perspective
Department of Computer Science（计算机科学系）；Brunel University of London（布鲁内尔大学）；Hanoi University of Science and Technology（河内科学技术大学）；Faculty of Electrical and Electronic Engineering School of Engineering（电气电子工程系工程学院）；Phenikaa University（Phenikaa大学）；Greenwich Vietnam FPT University（格林威治越南FPT大学）；College of Engineering（工程学院）；Shibaura Institute of Technology（Shibaura技术学院）
Membership Privacy Risks of Sharpness Aware Minimization
Department of Computer Science（计算机科学系）；Purdue University（普渡大学）；Delft University of Technology（代尔夫特理工大学）；Department of Industrial and Systems Engineering（工业与系统工程系）；University of Southern California（南加州大学）
Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Elmore Family School of Electrical and Computer Engineering, Purdue University（普渡大学埃尔默电气与计算机工程学院）
Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Models
Department of Mathematics, University of Missouri（数学系，密苏里大学）；Halıcıoğlu Data Science Institute, University of California San Diego（哈利奇奥数据科学研究所，加州大学圣地亚哥分校）
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Department of Computer Science（计算机科学系）；Department of Mathematics（数学系）；Mathematical Institute（数学研究所）；University of Pisa（比萨大学）；Imperial College London（伦敦帝国学院）；University of Oxford（牛津大学）；Section of Mathematics and Artificial Intelligence（数学与人工智能系）；Max Planck Institute of Molecular Cell Biology and Genetics（马克斯·普朗克分子细胞生物学和遗传学研究所）；Center for Systems Biology（系统生物学中心）；Faculty of Mathematics（数学系）；TU Dresden（德累斯顿技术大学）
Understanding and Mitigating Memorization in Diffusion Models for Tabular Data
Department of Computer；Data Sciences, Case Western Reserve University, Cleveland, USA；Department of Computer Science \& Engineering, Texas A\&M University, College Station, USA；Department of Biochemistry, Case Western Reserve University, Cleveland, USA；Center for RNA Science；Therapeutics, Case Western Reserve University, Cleveland, USA；Department of Biomedical Engineering, Case Western Reserve University, Cleveland, USA
Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data
King Abdullah University of Science and Technology（卡塔尔国王 Abdullah 科学与技术大学）；Renmin University of China（中国人民大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Beijing Institute of Technology（北京理工大学）
Exploring Model Dynamics for Accumulative Poisoning Discovery
Hong Kong Baptist University（香港浸会大学）；Alibaba Group（阿里巴巴集团）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The University of Sydney（悉尼大学）
KTH Royal Institute of Technology（皇家理工学院）；Nordita, Nordic Institute for Theoretical Physics（北欧理论物理研究所）；Hasso Plattner Institute, University of Potsdam（波茨坦大学哈索 Plattner 研究院）；Trinity College Dublin（都柏林三一学院）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；The Hong Kong University of Science（香港科学大学）；Mohamed bin Zayed University of Artificial Intelligence（莫莫丁·本·扎耶德人工智能大学）
Unconditional Latent Diffusion Models Memorize Patient Imaging Data: Implications for Openly Sharing Synthetic Data
Heidelberg University Hospital（海德堡大学医院）；AI Health Innovation Cluster (AIH)（AI健康创新集群）；German Centre for Cardiovascular Research (DZHK)（德国心血管研究中心）；University Medical Center Mannheim（曼海姆大学医学中心）；University Hospital Aachen（亚琛大学医院）；University Hospital Würzburg（维尔茨堡大学医院）；University Medical Center Mainz（美因茨大学医学中心）；TUD Dresden University of Technology（德累斯顿工业大学）；University Hospital Dresden（德累斯顿大学医院）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Department of Computer Science, Rensselaer Polytechnic Institute ( RPI )（里士满理工学院计算机科学系）；Donders Institute for Brain, Cognition, and Behaviour, Radboud University（拉布德大学脑科学与行为研究所）；Laboratoire de Physique Théorique et Modélisation, CY Cergy Paris Université（巴黎中央理工学院物理理论与建模实验室）
Link prediction on multi-relational graphs from an influence propagation perspective
School of Information Science and Technology, Yunnan Normal University（云南师范大学信息科学与技术学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）；School of Computer Science and Engineering, The University of New South Wales（新南威尔士大学计算机科学与工程学院）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；School of Computer and Mathematical Sciences, Adelaide University（阿德莱德大学计算机与数学科学学院）
Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective
School of Information Science and Technology, USTC and Institute of Artificial Intelligence, HCNSC（信息科学与技术学院，中国科学技术大学及人工智能研究所）；Department of Computer Science and Engineering, SUNY at Buffalo（计算机科学与工程系，纽约州立大学布法罗分校）
Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective
Department of Engineering, University of Cambridge（剑桥大学工程系）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Peng Cheng Laboratory（鹏城实验室）
