Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at https://github.com/molereddy/Alternate-Preference-Optimization.
Attention Smoothing Is All You Need For Unlearning
Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；Institute for AI and Data Science, Wayne State University（韦恩州立大学人工智能与数据科学研究所）
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机科学学院）；Shandong University（山东大学）；New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
针对 LLM 遗忘受知识表达形式制约的问题，本文提出 ORT 基准量化形式依赖偏差，并给出免训练的 ROCR 方法，通过秩一概念重定向实现更有效且自然的遗忘。
University of Virginia（弗吉尼亚大学）；EleutherAI；University of Exeter（埃克塞特大学）；Harvard Medical School（哈佛医学院）；University of Oxford（牛津大学）；Thomson-Reuters Foundational Research（汤森路透基础研究部）；Mass General Brigham（麻省总医院布里格姆医疗系统）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）
Quark: Controllable Text Generation with Reinforced Unlearning
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）
大规模的语言模型经常学习与用户期望不一致的行为。生成的文本可能包含攻击性或有毒语言，包含大量重复，或者与用户期望的不同情绪。我们考虑的任务是通过微调语言模型的信号不做什么来消除这些错位。我们引入了量化奖励约束（Quark），这是一种优化奖励函数的算法，该函数量化了（不）想要的属性，同时不会偏离原始模型太远。Quark在（i）收集具有当前语言模型的样本，（ii）基于奖励将它们分类为分位数，每个分位数由预先添加到语言模型的输入的奖励令牌标识，以及（iii）在来自每个分位数的以其奖励令牌为条件的样本上使用标准语言建模损失，同时通过KL发散惩罚保持在原始语言模型附近。通过在生成时以高奖励令牌为条件，该模型生成的文本显示出较少的不需要的属性。对于消除毒性，消极情绪和重复，我们的实验表明Quark优于强基线和最先进的强化学习方法，如PPO（Schulman et al. 2017），同时仅依赖于标准语言建模原语。
