CodeUnlearn：使用离散概念在语言模型中摊还式零样本机器遗忘
CodeUnlearn: Amortized Zero-Shot Machine Unlearning in Language Models Using Discrete Concept
AI总结：
针对LLM难以有效擦除特定数据及上下文的问题，提出利用码本特征和稀疏自编码器分解激活空间的摊还式零样本遗忘方法，在保持模型性能的同时实现特定主题遗忘。
摘要：
大型语言模型（LLMs）提供了跨领域的广泛知识，但它们可能无意中记住了敏感、未经授权或恶意的数据，例如医疗和金融领域的个人信息。机器遗忘方法旨在通过训练后从模型中移除特定信息来解决此问题。然而，由于LLMs复杂、密集且连续的特性，当前的方法需要额外的模型训练，或难以有效擦除特定数据点及其关联上下文。在本研究中，我们提出了一种使用码本特征和稀疏自编码器（SAEs）的新型摊还式遗忘方法。通过利用瓶颈结构来分解激活空间并调节信息流，我们的方法在保持模型于无关数据上性能的同时，高效地遗忘了目标信息。据我们所知，这是首个在LLM中成功实现具有上下文相关性的特定主题遗忘的工作，标志着向机器遗忘的现实应用迈出了重要一步。
英文摘要：
Large Language Models (LLMs) offer extensive knowledge across various domains, but they may inadvertently memorize sensitive, unauthorized, or malicious data, such as personal information in the medical and financial sectors. Machine unlearning methods aim to remove specific information from models after training to address this. However, current approaches require additional model training or struggle to effectively erase particular data points and their associated context due to LLMs' complex, dense, and continuous nature. In this study, we propose a novel amortized unlearning approach using codebook features and Sparse Autoencoders (SAEs). By leveraging a bottleneck to decompose the activation space and regulate information flow, our method efficiently unlearns targeted information while preserving the model's performance on unrelated data. To the best of our knowledge, this is the first work that successfully enables unlearning specific topics with contextual relevance in an LLM, marking a significant step towards real-world applications of machine unlearning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
