发表机构
Purdue University（普渡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出 Unmerge 算法，通过任务算术将机器遗忘视为逆操作，利用低秩遗忘基分解和三个优化目标，在保持高效的同时显著提升遗忘质量，并适用于多种模型规模。
AI 中文摘要
近似机器遗忘旨在无需完全重新训练的情况下，从已训练模型中移除遗忘集的影响。现有的基于梯度的方法需要依赖数据的超参数搜索，在遗忘知识与保留知识纠缠时难以处理，并且对网络中遗忘实际发生的位置提供很少的洞察。我们将遗忘问题重新表述为任务算术的视角：如果微调产生一个合并的任务向量 $\ au_m$，它结合了在遗忘集和保留集上的学习，那么遗忘就是逆操作，即减去一个学习到的遗忘分量 $\ au_F$ 以恢复保留任务向量 $\ au_R$。遗忘信号是集中的：在每一层，遗忘激活位于由少数主导方向张成的子空间中，因此我们在一个低秩遗忘基中分解 $\ au_F$，该基在小的尾部特征值残差内是忠实的，并限制了校正对保留的扰动程度。然后我们优化三个直观的目标（在遗忘跨度内匹配合并向量、抑制泄漏到保留跨度、以及限制校正大小），这些目标在激活空间中可证明地限制遗忘泄漏和保留损伤。由此产生的算法 Unmerge 快速且强大：在 CIFAR-100 和 Tiny ImageNet 上使用 ResNet-50 进行类别级遗忘时，它相对于运行时间相当的基线将 Tug-of-War 提高了高达约 24%，相对于运行速度慢约 5 倍的更强基线提高了高达约 18%，将成员推断暴露保持在重新训练的水平，并缩小了与重新训练模型的特征分布差距，而重新标记方法则使遗忘特征清晰可分离。进一步研究表明，Unmerge 也适用于 ViT-S/16，并可扩展到 Llama-3.2-3B。驱动该算法的逐层基几何结构也可作为逐层诊断工具，用于判断遗忘在何时何地变得结构性困难。
英文摘要
Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector $τ_m$ that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component $τ_F$ to recover the retain task vector $τ_R$. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize $τ_F$ in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.
