arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.19732cs.LGcs.AI
收藏

基于统一梯度与剩余几何增强的机器遗忘方法

Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement

Shanghai Jiao Tong University(上海交通大学)

Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, Xiaolin Huang

AI总结:

本文提出一种统一的基于梯度的机器遗忘方法,通过将遗忘更新嵌入由剩余数据二阶Hessian构建的流形几何中,并设计快慢参数更新策略隐式逼近显著遗忘方向,在分类与生成任务上以极低计算成本实现高效遗忘。

摘要:

机器遗忘(MU)的出现旨在增强深度神经网络的隐私性与可信度。近似MU是大规模模型的一种实用方法。我们对近似MU的研究始于在参数邻域内确定使输出Kullback-Leibler散度相对于精确MU最小化的最速下降方向。该探测方向可分解为三个组成部分:加权遗忘梯度上升、微调保留梯度下降以及权重显著性矩阵。这种源自欧几里得度量的分解涵盖了大多数现有的基于梯度的MU方法。然而,由于忽略了输出概率空间的几何结构,固守欧几里得空间可能导致次优的迭代轨迹。我们建议将遗忘更新嵌入到由剩余几何所呈现的流形中,并引入来自剩余数据的二阶Hessian信息。这有助于防止有效遗忘干扰保留性能。然而,为大规模模型计算二阶Hessian是难以处理的。为了高效利用Hessian调制的优势,我们提出了一种快慢参数更新策略,以隐式逼近最新的显著遗忘方向。由于不受特定模态约束的限制,我们的方法可适配计算机视觉遗忘任务,包括分类和生成。大量实验验证了我们的有效性和高效性。值得注意的是,我们的方法在使用DiT的ImageNet上成功实现了类别遗忘,并在CIFAR-10上使用DDPM仅用50步即遗忘一个类别,而先前方法需要数千步。

英文摘要:

Machine unlearning (MU) has emerged to enhance the privacy and trustworthiness of deep neural networks. Approximate MU is a practical method for large-scale models. Our investigation into approximate MU starts with identifying the steepest descent direction, minimizing the output Kullback-Leibler divergence to exact MU inside a parameters' neighborhood. This probed direction decomposes into three components: weighted forgetting gradient ascent, fine-tuning retaining gradient descent, and a weight saliency matrix. Such decomposition derived from Euclidean metric encompasses most existing gradient-based MU methods. Nevertheless, adhering to Euclidean space may result in sub-optimal iterative trajectories due to the overlooked geometric structure of the output probability space. We suggest embedding the unlearning update into a manifold rendered by the remaining geometry, incorporating second-order Hessian from the remaining data. It helps prevent effective unlearning from interfering with the retained performance. However, computing the second-order Hessian for large-scale models is intractable. To efficiently leverage the benefits of Hessian modulation, we propose a fast-slow parameter update strategy to implicitly approximate the up-to-date salient unlearning direction. Free from specific modal constraints, our approach is adaptable across computer vision unlearning tasks, including classification and generation. Extensive experiments validate our efficacy and efficiency. Notably, our method successfully performs class-forgetting on ImageNet using DiT and forgets a class on CIFAR-10 using DDPM in just 50 steps, compared to thousands of steps required by previous methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑