Machine unlearning (MU) has emerged to enhance the privacy and trustworthiness of deep neural networks. Approximate MU is a practical method for large-scale models. Our investigation into approximate MU starts with identifying the steepest descent direction, minimizing the output Kullback-Leibler divergence to exact MU inside a parameters' neighborhood. This probed direction decomposes into three components: weighted forgetting gradient ascent, fine-tuning retaining gradient descent, and a weight saliency matrix. Such decomposition derived from Euclidean metric encompasses most existing gradient-based MU methods. Nevertheless, adhering to Euclidean space may result in sub-optimal iterative trajectories due to the overlooked geometric structure of the output probability space. We suggest embedding the unlearning update into a manifold rendered by the remaining geometry, incorporating second-order Hessian from the remaining data. It helps prevent effective unlearning from interfering with the retained performance. However, computing the second-order Hessian for large-scale models is intractable. To efficiently leverage the benefits of Hessian modulation, we propose a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction. Free from specific modal constraints, our approach is adaptable across computer vision unlearning tasks, including classification and generation. Extensive experiments validate our efficacy and efficiency. Notably, our method successfully performs class-forgetting on ImageNet using DiT and forgets a class on CIFAR-10 using DDPM in just 50 steps, compared to thousands of steps required by previous methods.
Machine Unlearning with Minimal Gradient Dependence for High Unlearning Ratios
School of Information, Renmin University of China(中国人民大学信息学院);School of Computer Science and Big Data, Minjiang University(闽江学院计算机与大数据学院);RMIT University(皇家墨尔本理工大学)
Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Institute of Artificial Intelligence, Beihang University(北航人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(北京未来区块链与隐私计算先进创新中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Amsterdam(阿姆斯特丹大学);Tsinghua University(清华大学);YanTron Technology Co.Ltd(燕tron技术有限公司)
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Department of Electrical and Computer Engineering(电气与计算机工程系);Georgia Institute of Technology(佐治亚理工学院);Department of Mathematics(数学系);Massachusetts Institute of Technology(麻省理工学院)
SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);University of California San Diego(加州大学圣地亚哥分校)
FedCARE: Federated Unlearning with Conflict-Aware Projection and Relearning-Resistant Recovery
School of Cyber Engineering, Xidian University(电子科技大学信息工程学院);RMIT University(皇家墨尔本理工大学);Commonwealth Scientific and Industrial Research Organisation (CSIRO)(澳大利亚联邦科学与工业研究组织);UNSW Sydney(新南威尔士大学悉尼分校);University of California, Davis(加州大学戴维斯分校)
Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks
Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, China(交通物联网湖北重点实验室,计算机科学与人工智能学院,武汉理工大学,中国);College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院,南洋理工大学,新加坡)
Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning
Sydney AI Centre, The University of Sydney(悉尼人工智能中心,悉尼大学);Hong Kong Baptist University(香港 Baptist 大学);RIKEN AIP(日本理化学研究所AIP);University of Technology Sydney(悉尼技术大学)
Controllable Machine Unlearning via Gradient Pivoting
Artificial Intelligence Graduate School, UNIST, South Korea(人工智能研究生院,UNIST,韩国);Department of Industrial Engineering, UNIST, South Korea(工业工程系,UNIST,韩国)
Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning
DIBRIS – Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova(热那亚大学信息、生物工程、机器人与系统工程系(DIBRIS));Centre for Defense Higher Studies (CASD)(国防高等研究中心(CASD))
Targeted Unlearning Using Perturbed Sign Gradient Methods With Applications On Medical Images
Department of Biomedical Engineering(生物医学工程系);Department of Ophthalmology(眼科学系);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Department of Computer Science(计算机科学系);Computer Science and Engineering(计算机科学与工程);Pohang University of Science and Technology, South Korea(韩国釜山科学技术大学);Department of Plastic and Reconstructive Surgery(整形外科与重建外科系)
Machine Unlearning of Pre-trained Large Language Models
University of Virginia(弗吉尼亚大学);Georgia Institute of Technology(佐治亚理工学院);The Hong Kong Polytechnic University(香港理工大学);University of Melbourne(墨尔本大学);Carnegie Mellon University(卡内基梅隆大学)
The Measure of Deception: An Analysis of Data Forging in Machine Unlearning
Department of Mathematics, UC San Diego(加州大学圣地亚哥分校数学系);Department of Mathematics and Halıcıoglu Data Science Institute, UC San Diego(加州大学圣地亚哥分校数学系与哈勒吉奥卢数据科学研究所)
Electrical Engineering and Computer Science, University of California, Merced, CA 95343, USA(电气工程与计算机科学系,加州大学默塞德分校);Department of Applied Mathematics, University of California, Merced, CA 95343, USA(应用数学系,加州大学默塞德分校);Department of Innovation and Research, North Carolina State University, Raleigh, NC, USA(创新与研究系,北卡罗来纳州立大学);Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA(自动化(MESA)实验室,机械工程系,工程学院,加州大学默塞德分校)