稳定遗忘：基础模型中的有界参数高效反遗忘
Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
AI总结：
本文提出稳定反遗忘方法，通过限制MLP适配器的权重动态，解决基础模型中反遗忘的不稳定问题，并在多个基准测试中验证其有效性。
摘要：
在基础模型（如语言和视觉转换器）中进行机器反遗忘对于隐私和安全至关重要；然而，现有方法不稳定且不可靠。一种广泛使用的策略是通过梯度下降保留数据，同时对遗忘数据进行梯度上升。当与交叉熵结合时，此过程可能导致权重和梯度的无界增长，从而降低遗忘和保留质量。我们提供了一个理论框架，解释了这一失败的原因，展示了如何通过上升破坏转换器前馈MLP层的优化。受这一见解的启发，我们提出了*有界参数高效反遗忘*，通过应用有界函数到MLP适配器，稳定LoRA基于的微调。此方法控制了上升过程中的权重动态，从而实现可靠的收敛。我们在CIFAR-100上的视觉转换器类别删除上验证了该方法，其中GD+Sine是唯一能实现高遗忘质量和模型效用的方法，适用于ViT-B/16、ViT-L/14和DeiT-S架构。此外，在语言模型基准测试（TOFU、TDEC、MUSE）上展示了其通用性，适用于从22M到8B参数的架构，实现了改进的遗忘同时保持效用。
英文摘要：
Machine unlearning in foundation models (e.g., language and vision transformers) is essential for privacy and safety; however, existing approaches are unstable and unreliable. A widely used strategy, the gradient difference method, applies gradient descent to retained data while performing gradient ascent on forgotten data. When combined with cross-entropy, this procedure can trigger the unbounded growth of weights and gradients, degrading both forgetting and retention. We provide a theoretical framework that explains this failure by showing how ascent destabilizes optimization in transformer feedforward MLP layers. Guided by this insight, we propose *Bounded Parameter-Efficient Unlearning*, which stabilizes LoRA-based fine-tuning by applying bounded functions to MLP adapters. This controls the weight dynamics during ascent and enables reliable convergence. We validate the approach on Vision Transformer class deletion on CIFAR-100, where GD+Sine is the only evaluated method to achieve both high forget quality and model utility across ViT-B/16, ViT-L/14, and DeiT-S architectures, and demonstrate generality on language-model benchmarks (TOFU, TDEC, MUSE) across architectures from 22M to 8B parameters, achieving improved forgetting while preserving utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
