Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark.
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Harbin Institute of Technology(哈尔滨工业大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Unlimidata Ltd(Unlimidata有限公司);The Chinese University of Hong Kong(香港中文大学);University of Calabria(卡拉布里亚大学)
Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
Department of Mathematics, University of California Davis, USA(加州大学戴维斯分校数学系);SLAC National Accelerator Laboratory, Stanford University, USA(斯坦福大学SLAC国家加速器实验室);Department of Computer Science, University of California Davis, USA(加州大学戴维斯分校计算机科学系)
$\textbf{AGT$^{AO}$}$: Robust and Stabilized LLM Unlearning via Adversarial Gating Training with Adaptive Orthogonality
School of Computer Science and Technology, Xi’an Jiaotong University, China(西安交通大学计算机科学与技术学院);MOE KLINNS Lab, Xi’an Jiaotong University, China(西安交通大学MOE KLINNS实验室);Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China(陕西省大数据知识工程重点实验室)
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Harbin Institute of Technology(哈尔滨工业大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));BIAI, ZJUT & D5Data.ai(BIAI、浙江工业大学及D5Data.ai);National University of Singapore(新加坡国立大学)
Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
The Hong Kong Polytechnic University(香港理工大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Santa Cruz(加州大学圣克ruz分校);Huawei Technologies(华为技术有限公司);Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU(未来智能系统中的隐私与安全技术研究中心,PolyU)
UniErase: Towards Balanced and Precise Unlearning in Language Models
University of Science and Technology of China(中国科学技术大学);University of the Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);A*STAR(科技研究局);Zhejiang University(浙江大学)
ReLearn: Unlearning via Learning for Large Language Models
Zhejiang University(浙江大学);Xiamen University(厦门大学);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);National University of Singapore(新加坡国立大学);NUS-NCS Joint Lab(新加坡国立大学-新科系统联合实验室)
针对现有大模型遗忘方法破坏文本连贯性与评估指标不足的问题,提出数据增强微调流水线 ReLearn 及包含 KFR、KRR 和 LS 的综合评估框架,实现了高效目标遗忘并保持高质量生成。
Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)