Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery
Institute of Artificial Intelligence, Beihang University(北航人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(北京未来区块链与隐私计算先进创新中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Amsterdam(阿姆斯特丹大学);Tsinghua University(清华大学);YanTron Technology Co.Ltd(燕tron技术有限公司)
Machine unlearning (MU) aims to efficiently remove sensitive or harmful memory from a pre-trained model. The key challenge is to balance the potential tradeoff between unlearning efficacy and utility preservation, which involves forgetting undesirable information as defined while maintaining the model's original performance. One potential way to tackle this problem is to use multi-objective optimization to jointly optimize both the unlearning and utility preservation objectives. However, existing multi-objective methods only guarantee finding a Pareto-optimal solution without fine-grained control, which causes under-optimization of the unlearning objective. To this end, we first model MU as a constrained optimization problem, that is, optimizing the unlearning objective under the constraint of a bounded increase for utility loss. We then show that solving this optimization problem is equivalent to unilateral gradient surgery on the unlearning objective. To resolve the additional computational cost brought by gradient surgery, we propose an implicit gradient surgery method, which approximates the solution to the aforementioned constrained optimization problem via only one backpropagation, thereby achieving efficient utility-preserving MU. Theoretically, we provide a tight convergence analysis of the algorithm. Empirically, our extensive experiments show that the proposed algorithm achieves better tradeoff results than existing baselines. Codes are available at https://github.com/anseryuer/EUPMU-Efficient-Utility-Preserving-Machine-Unlearning.
Certified Unlearning in Decentralized Federated Learning
School of Computer Science and Technology, Shandong University, Qingdao, China(山东大学计算机科学与技术学院);School of Electrical Engineering and Computer Science, TU Berlin, Berlin, Germany(柏林技术大学电气工程与计算机科学学院);College of Computer Science and Technology, Ocean University of China, Qingdao, China(中国海洋大学计算机科学与技术学院)
Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient
Southeast University(东南大学);Tsinghua University(清华大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of California, Berkeley(加州大学伯克利分校);Nanyang Technological University(南洋理工大学)
Federated Unlearning with Gradient Descent and Conflict Mitigation
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Guangzhou University(广州大学);The Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院);University of Michigan(密歇根大学);Shenzhen Research Institute of Big Data(深圳大数据研究院)