Standard large language model training can create models that produce outputs their trainer deems unacceptable in deployment. The probability of these outputs can be reduced using methods such as LLM unlearning. However, unlearning a set of data (called the forget set) can degrade model performance on other distributions where the trainer wants to retain the model's behavior. To improve this trade-off, we demonstrate that using the forget set to compute only a few uphill Gauss-Newton steps provides a conceptually simple, state-of-the-art unlearning approach for LLMs. While Gauss-Newton steps adapt Newton's method to non-linear models, it is non-trivial to efficiently and accurately compute such steps for LLMs. Hence, our approach crucially relies on parametric Hessian approximations such as Kronecker-Factored Approximate Curvature (K-FAC). We call this combined approach K-FADE (K-FAC for Distribution Erasure). Our evaluation on the WMDP and ToFU benchmarks demonstrates that K-FADE suppresses outputs from the forget set and approximates, in output space, the results of retraining without the forget set. Critically, our method does this while altering the outputs on the retain set less than previous methods. This is because K-FADE transforms a constraint on the model's outputs across the entire retain set into a constraint on the model's weights, allowing the algorithm to minimally change the model's behavior on the retain set at each step. Moreover, the unlearning updates computed by K-FADE can be reapplied later if the model undergoes further training, allowing unlearning to be cheaply maintained.
Privacy laws and regulations enforce data-driven systems, e.g., recommender systems, to erase the data that concern individuals. As machine learning models potentially memorize the training data, data erasure should also unlearn the data lineage in models, which raises increasing interest in the problem of Machine Unlearning (MU). However, existing MU methods cannot be directly applied into recommendation. The basic idea of most recommender systems is collaborative filtering, but existing MU methods ignore the collaborative information across users and items. In this paper, we propose a genera…
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智慧城市物联网国家重点实验室）；Mila – Québec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；RIKEN AIP（日本理化学研究所先进智能研究中心）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Google Research（谷歌研究院）
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；MPI, Tübingen（蒂宾根马克斯·普朗克研究所）；Oregon State University（俄勒冈州立大学）；Google DeepMind（谷歌DeepMind）；Princeton University（普林斯顿大学）
Gaussian Certified Unlearning in High Dimensions: A Hypothesis Testing Approach
Department of Operations Research and Financial Engineering, Princeton University（普林斯顿大学运筹学与金融工程系）；Department of Statistics, The Ohio State University（俄亥俄州立大学统计学系）；Department of Statistics, Columbia University（哥伦比亚大学统计学系）；Department of Electrical and Computer Engineering and Department of Operations Research and Financial Engineering, Princeton University（普林斯顿大学电气工程系和运筹学与金融工程系）
School of Computing and Information Systems（计算与信息系统学院）；University of Melbourne（墨尔本大学）；Department of Computer Science（计算机科学系）；Amherst College（阿默斯特学院）
The right to erasure requires removal of a user's information from data held by organizations, with rigorous interpretations extending to downstream products such as learned models. Retraining from scratch with the particular user's data omitted fully removes its influence on the resulting model, but comes with a high computational cost. Machine "unlearning" mitigates the cost incurred by full retraining: instead, models are updated incrementally, possibly only requiring retraining when approximation errors accumulate. Rapid progress has been made towards privacy guarantees on the indistinguis…
Attention Smoothing Is All You Need For Unlearning
Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；Institute for AI and Data Science, Wayne State University（韦恩州立大学人工智能与数据科学研究所）
