持久的遗忘：大语言模型中保留效用、鲁棒且几乎不可逆的遗忘
Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
AI总结：
针对现有大语言模型遗忘方法效果不足的问题，提出基于Jensen-Shannon散度的JensUn方法，实现更优的遗忘-效用权衡与抗再学习鲁棒性，同时构建LKF数据集并提出改进评估框架，更精准地衡量遗忘方法效果。
摘要：
大语言模型（LLM）中的遗忘（unlearning）指从预训练模型中精准移除特定信息，这对删除预训练过程中获取的隐私数据或有害知识、保障LLM安全至关重要。但现有遗忘方法经全面评估后往往存在不足。为此，我们提出JensUn方法，采用Jensen-Shannon散度作为遗忘集和保留集的训练目标，相比常用损失函数，能带来更稳定有效的遗忘动态。大量实验表明，JensUn在遗忘效果与效用保留的权衡上优于同类方法，还对良性再学习表现出强鲁棒性。此外，为实现精准的遗忘评估，我们构建了LKF（小众事实精选数据集），提供贴近真实的遗忘场景。最后，为全面测试遗忘方法，我们提出两点方案：一是用LLM作为语义评判器替代标准ROUGE分数，二是针对多种释义和输入格式开展最坏情况遗忘评估。这套改进的评估框架显示，许多现有方法的实际效果不如之前认为的那样好。
英文摘要：
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private data or harmful knowledge acquired during pre-training. However, existing unlearning methods often fall short when subjected to thorough evaluation. To overcome this, we introduce JensUn, where we leverage the Jensen-Shannon Divergence as the training objective for both forget and retain sets for more stable and effective unlearning dynamics compared to commonly used loss functions. In extensive experiments, JensUn achieves better forget-utility trade-off than competing methods, and even demonstrates strong resilience to benign relearning. Additionally, for a precise unlearning evaluation, we introduce LKF, a curated dataset of lesser-known facts that provides a realistic unlearning scenario. Finally, to comprehensively test unlearning methods, we propose (i) employing an LLM as semantic judge instead of the standard ROUGE score, and (ii) using worst-case unlearning evaluation over various paraphrases and input formats. Our improved evaluation framework reveals that many existing methods are less effective than previously thought.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
