Large language models (LLMs) have advanced to encompass extensive knowledge across diverse domains. Yet controlling what a large language model should not know is important for ensuring alignment and thus safe use. However, accurately and efficiently unlearning knowledge from an LLM remains challenging due to the potential collateral damage caused by the fuzzy boundary between retention and forgetting, and the large computational requirements for optimization across state-of-the-art models with hundreds of billions of parameters. In this work, we present \textbf{Embedding-COrrupted (ECO) Prompts}, a lightweight unlearning framework for large language models to address both the challenges of knowledge entanglement and unlearning efficiency. Instead of relying on the LLM itself to unlearn, we enforce an unlearned state during inference by employing a prompt classifier to identify and safeguard prompts to forget. We learn corruptions added to prompt embeddings via zeroth order optimization toward the unlearning objective offline and corrupt prompts flagged by the classifier during inference. We find that these embedding-corrupted prompts not only lead to desirable outputs that satisfy the unlearning objective but also closely approximate the output from a model that has never been trained on the data intended for forgetting. Through extensive experiments on unlearning, we demonstrate the superiority of our method in achieving promising unlearning at \textit{nearly zero side effects} in general domains and domains closely related to the unlearned ones. Additionally, we highlight the scalability of our method to 100 LLMs, ranging from 0.5B to 236B parameters, incurring no additional cost as the number of parameters increases. We have made our code publicly available at \url{https://github.com/chrisliu298/llm-unlearn-eco}.
Attention Smoothing Is All You Need For Unlearning
Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；Institute for AI and Data Science, Wayne State University（韦恩州立大学人工智能与数据科学研究所）
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Technical University of Munich（慕尼黑技术大学）；Siemens AG（西门子股份公司）；University of Oxford（牛津大学）
Consistency-Aware Editing for Entity-level Unlearning in Language Models
Shanxi University, China（山西大学）；Cardiff University, UK（卡地夫大学）；Singapore University of Technology and Design, Singapore（新加坡科技设计大学）；ILCC, School of Informatics, University of Edinburgh, Edinburgh, UK（爱丁堡大学信息学院ILCC）
CBD: API-Only LLM Black-Box Unlearning through Controlled Behavioral Divergence
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算高精尖创新中心）；Department of Electronic and Electrical Engineering, Sungkyunkwan University（成均馆大学电子与电气工程系）
State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门大学智慧城市物联网国家重点实验室）；Mila – Québec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；RIKEN AIP（日本理化学研究所先进智能研究中心）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Google Research（谷歌研究院）
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；BIAI, ZJUT & D5Data.ai（BIAI、浙江工业大学及D5Data.ai）；National University of Singapore（新加坡国立大学）
Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models
School of Computing（计算机学院）；National University of Singapore（新加坡国立大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；College of Design and Engineering（设计与工程学院）；RespAI Lab（RespAI实验室）；School of Computer Engineering（计算机工程学院）；Institute for Infocomm Research（信息与通信研究所）；A*STAR
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Glasgow（格拉斯哥大学）
Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；Jianghuai Advanced Technology Center（江淮先进技术中心）；University of Technology Sydney（悉尼大学）
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
Department of Computer Science（计算机科学系）；University of North Carolina at Chapel Hill（北卡罗来纳大学 Chapel Hill 分校）；School of Artificial Intelligence and Data Science（人工智能与数据科学学院）；University of Science and Technology of China（中国科学技术大学）
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）；Sichuan University（四川大学）；Zhejiang Normal University（浙江师范大学）
