SAU: 通过梯度掩码和重要性再分配实现LLM的稀疏性感知反学习
SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution
AI总结：
本文提出SAU方法，通过梯度掩码和重要性再分配解决稀疏模型中反学习效果下降问题，提升模型遗忘能力并保持实用性。
摘要：
大型语言模型（LLMs）在训练过程中不可避免地会记忆敏感信息，带来重大隐私风险。反学习作为一种去除此类信息而不需完整重训练的解决方案逐渐兴起。然而，现有方法针对密集模型设计，忽略了模型稀疏化，这是高效LLM部署的关键技术。我们发现稀疏模型中反学习效果显著下降。通过实证分析，我们发现这种下降是因为现有反学习方法需要更新所有参数，而稀疏化将大量权重设为零，从根本上限制了模型的遗忘能力。为解决这一挑战，我们提出了稀疏性感知反学习（SAU），通过梯度掩码将更新重定向到存活权重，结合重要性感知再分配来补偿剪枝参数。广泛实验表明，SAU在稀疏LLM上显著优于现有方法，实现有效的遗忘同时保持模型实用性。
英文摘要：
Large Language Models (LLMs) inevitably memorize sensitive information during training, posing significant privacy risks. Machine unlearning has emerged as a promising solution to selectively remove such information without full retraining. However, existing methods are designed for dense models and overlook model sparsification, an essential technique for efficient LLM deployment. We find that unlearning effectiveness degrades substantially on sparse models. Through empirical analysis, we reveal that this degradation occurs because existing unlearning methods require updating all parameters, yet sparsification prunes substantial weights to zero, fundamentally limiting the model's forgetting capacity. To address this challenge, we propose Sparsity-Aware Unlearning (SAU), which decouples unlearning from sparsification objectives through gradient masking that redirects updates to surviving weights, combined with importance-aware redistribution to compensate for pruned parameters. Extensive experiments demonstrate that SAU significantly outperforms existing methods on sparse LLMs, achieving effective forgetting while preserving model utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
