发表机构
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究通过模式连通性视角理解模型遗忘，引入MCU并评估，发现遗忘模型处于连通盆地，训练动态会影响，同一盆地模型隐私指标有别，遗忘非线性，近似遗忘方法与重新训练不同，基于MCU的集成可提升泛化和鲁棒性。
AI 中文摘要
模型遗忘旨在从训练好的模型中去除不需要的信息，而无需从头完全重新训练。尽管最近有进展，但对遗忘的损失景观和优化几何仍了解不足。本文通过模式连通性（即独立训练的模型在参数空间中常可由平滑低损失路径连接的现象）来研究模型遗忘。我们引入了“遗忘中的模式连通性”（MCU）并在一系列设置中进行评估，包括课程学习、二阶优化以及不同遗忘方法间的连通性。我们发现许多遗忘后的模型位于具有平滑保留/遗忘行为的连通盆地中，训练动态的变化可将解转移到不同盆地。MCU还表明同一盆地内的模型在隐私指标上可能有很大差异，且遗忘从原始模型到遗忘后模型是非线性进展的。此外，线性连通性表明大多数近似遗忘方法在机制上与重新训练不同。最后，基于MCU的集成可提高对重新学习攻击的泛化能力和鲁棒性，且MCU平滑度与遗忘难度相关。据我们所知，这是首次通过模式连通性视角对模型遗忘进行研究。
英文摘要
Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood. In this paper, we study machine unlearning through the lens of mode connectivity--the phenomenon that independently trained models can often be connected by smooth low-loss paths in parameter space. We introduce {\em mode connectivity in unlearning} (MCU) and evaluate it across a range of settings, including curriculum learning, second-order optimization, and connectivity across different unlearning methods. We find that many unlearned models lie in connected basins with smooth retain/forget behavior, while changes in training dynamics can move solutions into different basins. MCU also reveals that models within the same basin can differ substantially on privacy metrics, and that unlearning progresses nonlinearly from the original model to the unlearned model. In addition, linear connectivity suggests that most approximate unlearning methods are mechanistically distinct from retraining. Finally, MCU-based ensembling can improve generalization and robustness to relearning attacks, and MCU smoothness correlates with unlearning difficulty. To our knowledge, this is the first study of machine unlearning through the lens of mode connectivity.
CommentsThis work was intended as a replacement of arXiv:2504.06407 and any subsequent updates will appear there
