The dynamics of gradient-based training in neural networks often exhibit nontrivial structures; hence, understanding them remains a central challenge in theoretical machine learning. In particular, a concept of feature unlearning, in which a neural network progressively loses previously learned features over long training, has gained attention. In this study, we consider the infinite-width limit of a two-layer neural network updated with a large-batch stochastic gradient, then derive differential equations with different time scales, revealing the mechanism and conditions for feature unlearning to occur. Specifically, we utilize the fast-slow dynamics: while an alignment of first-layer weights develops rapidly, the second-layer weights develop slowly. The direction of a flow on a critical manifold, determined by the slow dynamics, decides whether feature unlearning occurs. We give numerical validation of the result, and derive theoretical grounding and scaling laws of the feature unlearning. Our results yield the following insights: (i) the strength of the primary nonlinear term in data induces the feature unlearning, and (ii) an initial scale of the second-layer weights mitigates the feature unlearning. Technically, our analysis utilizes Tensor Programs and the singular perturbation theory.
Descend or Rewind? Stochastic Gradient Descent Unlearning
Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL, United States（工程科学与应用数学系，西北大学，埃文斯顿，伊利诺伊州，美国）；Department of Industrial Engineering（工业工程系）；Applied Mathematics, Northwestern University, Evanston, IL, United States（应用数学，西北大学，埃文斯顿，伊利诺伊州，美国）
Federated Unlearning with Gradient Descent and Conflict Mitigation
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Guangzhou University（广州大学）；The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；University of Michigan（密歇根大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Department of Electrical and Computer Engineering（电气与计算机工程系）；Georgia Institute of Technology（佐治亚理工学院）；Department of Mathematics（数学系）；Massachusetts Institute of Technology（麻省理工学院）
On the Dynamics of Delayed Generalization in Linear Models
Department of XXX, University of YYY, Location, Country（YYY大学XXX系）；School of ZZZ, Institute of WWW, Location, Country（WWW研究所ZZZ学院）；Indian Institute of Technology Madras, Chennai（马德拉斯印度理工学院）
Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles
Department of Electrical Engineering and Electronics, University of Liverpool（利物浦大学电子工程与电子系）；Department of Computer and Information Science, Linköping University（林雪平大学计算机与信息科学系）
Machine Unlearning of Pre-trained Large Language Models
University of Virginia（弗吉尼亚大学）；Georgia Institute of Technology（佐治亚理工学院）；The Hong Kong Polytechnic University（香港理工大学）；University of Melbourne（墨尔本大学）；Carnegie Mellon University（卡内基梅隆大学）
Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；Hong Kong Baptist University（香港 Baptist 大学）；RIKEN AIP（日本理化学研究所AIP）；University of Technology Sydney（悉尼技术大学）
Decoupling the Class Label and the Target Concept in Machine Unlearning
Hong Kong Baptist University（香港浸会大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能中心）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Tokyo（东京大学）
