抑制或删除:基于恢复的表示层面机器去学习分析
Suppression or Deletion: A Restoration-Based Representation-Level Analysis of Machine Unlearning
AI总结:
本文提出了一种基于恢复的分析框架,揭示了现有去学习方法在表示层面保留信息的风险,强调了对新评估标准的需求。
摘要:
随着预训练模型越来越多地在网络上传播,确保模型在请求时能够忘记或删除敏感、受版权保护或私人信息变得至关重要。机器去学习已被提出以解决这一挑战。然而,当前的去学习方法评估依赖于输出基于的指标,这些指标无法验证信息是否被完全删除或仅仅是被抑制在表示层面,其中抑制不足以实现真正的去学习。为了解决这一差距,我们提出了一种新的基于恢复的分析框架,利用稀疏自动编码器来识别中间层中的类别特定专家特征,并应用推理时间引导以定量区分抑制和删除。将我们的框架应用于12种主要的图像分类任务中的去学习方法,我们发现大多数方法在去学习信息的恢复率上表现很高,表明它们仅在决策边界层面抑制信息,而在中间表示中保留语义特征。值得注意的是,即使从预训练检查点重新训练也表现出高恢复率,揭示了从预训练中继承的稳健语义特征并未被重新训练所移除。这些结果表明,表示层面的保留对输出基于的指标来说存在重大风险,突显了对新去学习评估标准的需求。我们提出新的评估指南,优先考虑表示层面的验证,特别是在预训练模型时代的隐私关键应用中。
英文摘要:
As pretrained models are increasingly shared on the web, ensuring that models can forget or delete sensitive, copyrighted, or private information upon request has become crucial. Machine unlearning has been proposed to address this challenge. However, current evaluations for unlearning methods rely on output-based metrics, which cannot verify whether information is completely deleted or merely suppressed at the representation level, where suppression is insufficient for true unlearning. To address this gap, we propose a novel restoration-based analysis framework that uses Sparse Autoencoders to identify class-specific expert features in intermediate layers and applies inference-time steering to quantitatively distinguish between suppression and deletion. Applying our framework to 12 major unlearning methods in image classification tasks, we find that most methods achieve high restoration rates of unlearned information, indicating that they only suppress information at the decision-boundary level, while preserving semantic features in intermediate representations. Notably, even retraining from pretrained checkpoints shows high restoration, revealing that robust semantic features inherited from pretraining are not removed by retraining. These results demonstrate that representation-level retention poses significant risks overlooked by output-based metrics, highlighting the need for new unlearning evaluation criteria. We propose new evaluation guidelines that prioritize representation-level verification, especially for privacy-critical applications in the era of pre-trained models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。