As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents, and (2) it should retain the other knowledge that the LLM possesses, for which we assume access to a small number of retain documents. To achieve both goals, a mainstream class of LLM unlearning methods introduces an optimization framework with a combination of two objectives - maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetting. In this paper, we propose a novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims to achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned LLM by computing the logit difference between the target and the assistant LLMs. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM's overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility on average to achieve comparable forget quality. Our code will be publicly available at https://github.com/UCSB-NLP-Chang/ULD.
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Unlimidata Ltd（Unlimidata有限公司）；The Chinese University of Hong Kong（香港中文大学）；University of Calabria（卡拉布里亚大学）
Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Pittsburgh（匹兹堡大学）；University of Arizona（亚利桑那大学）；University of Central Florida（中佛罗里达大学）
Attention Smoothing Is All You Need For Unlearning
Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；Institute for AI and Data Science, Wayne State University（韦恩州立大学人工智能与数据科学研究所）
Improving LLM Unlearning Robustness via Random Perturbations
Japan Advanced Institute of Science and Technology（日本科学技术先进研究院）；VNU University of Engineering and Technology（越南工程与技术大学）；Monash University（莫纳什大学）；RIKEN（日本研究机构）
Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；University of Jinan（济南大学）；Pengcheng Laboratory（鹏城实验室）
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；University of Amsterdam（阿姆斯特丹大学）
GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；UC Santa Cruz（加州大学圣克鲁兹分校）；UIUC（伊利诺伊大学香槟分校）；Southeast University（东南大学）；BIAI, ZJUT（脑科学与类脑智能研究院，浙江大学）；D5Data.ai
Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
Department of Mathematics, University of California Davis, USA（加州大学戴维斯分校数学系）；SLAC National Accelerator Laboratory, Stanford University, USA（斯坦福大学SLAC国家加速器实验室）；Department of Computer Science, University of California Davis, USA（加州大学戴维斯分校计算机科学系）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
The Hong Kong Polytechnic University（香港理工大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Santa Cruz（加州大学圣克ruz分校）；Huawei Technologies（华为技术有限公司）；Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU（未来智能系统中的隐私与安全技术研究中心，PolyU）
