SEMU:用于高效机器遗忘的奇异值分解
SEMU: Singular Value Decomposition for Efficient Machine Unlearning
AI总结:
针对现有机器遗忘方法计算成本高、依赖原始数据集的问题,提出SEMU方法,通过奇异值分解创建低维投影,减少需修改的参数数量并摆脱对原始数据集的依赖,在提升效率的同时保证性能。
摘要:
近年来,尽管生成式基础模型的能力迅速提升,但防止有害和不安全行为的方法仍未得到充分发展。在人工智能安全的紧迫挑战中,机器遗忘(MU)对于满足即将出台的安全法规变得日益关键。大多数现有MU方法侧重于改变模型最重要的参数。然而,这些方法通常需要对模型的大部分进行微调,导致计算成本高且训练不稳定,而这些问题通常需要通过访问原始训练数据集来缓解。在这项工作中,我们利用奇异值分解(SVD)创建紧凑的低维投影,以实现对特定数据点的选择性遗忘,从而解决这些局限性。我们提出了用于高效机器遗忘的奇异值分解(SEMU),这是一种旨在从两个关键方面优化MU的新方法。首先,SEMU最大限度地减少了需要修改的模型参数数量,在仅对模型权重进行最小更改的情况下有效移除了不需要的知识。其次,SEMU消除了对原始训练数据集的依赖,无需额外数据即可保留模型先前获取的知识。大量实验表明,SEMU在数据使用和修改参数数量方面显著提高效率的同时,实现了具有竞争力的性能。
英文摘要:
While the capabilities of generative foundational models have advanced rapidly in recent years, methods to prevent harmful and unsafe behaviors remain underdeveloped. Among the pressing challenges in AI safety, machine unlearning (MU) has become increasingly critical to meet upcoming safety regulations. Most existing MU approaches focus on altering the most significant parameters of the model. However, these methods often require fine-tuning substantial portions of the model, resulting in high computational costs and training instabilities, which are typically mitigated by access to the original training dataset. In this work, we address these limitations by leveraging Singular Value Decomposition (SVD) to create a compact, low-dimensional projection that enables the selective forgetting of specific data points. We propose Singular Value Decomposition for Efficient Machine Unlearning (SEMU), a novel approach designed to optimize MU in two key aspects. First, SEMU minimizes the number of model parameters that need to be modified, effectively removing unwanted knowledge while making only minimal changes to the model's weights. Second, SEMU eliminates the dependency on the original training dataset, preserving the model's previously acquired knowledge without additional data requirements. Extensive experiments demonstrate that SEMU achieves competitive performance while significantly improving efficiency in terms of both data usage and the number of modified parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。