Unlearning methods have the potential to improve the privacy and safety of large language models (LLMs) by removing sensitive or harmful information post hoc. The LLM unlearning research community has increasingly turned toward empirical benchmarks to assess the effectiveness of such methods. In this paper, we find that existing benchmarks provide an overly optimistic and potentially misleading view on the effectiveness of candidate unlearning methods. By introducing simple, benign modifications to a number of popular benchmarks, we expose instances where supposedly unlearned information remains accessible, or where the unlearning process has degraded the model's performance on retained information to a much greater extent than indicated by the original benchmark. We identify that existing benchmarks are particularly vulnerable to modifications that introduce even loose dependencies between the forget and retain information. Further, we show that ambiguity in unlearning targets in existing benchmarks can easily lead to the design of methods that overfit to the given test queries. Based on our findings, we urge the community to be cautious when interpreting benchmark results as reliable measures of progress, and we provide several recommendations to guide future LLM unlearning research.
Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem
Shanghai University of Finance and Economics(上海金融学院);Alibaba Group(阿里巴巴集团);Southern University of Science and Technology(南方科技大学);Beihang University(北航);MoE Key Laboratory of Interdisciplinary Research of Computation and Economics(计算与经济交叉学科研究教育部重点实验室)
Safety Alignment via Constrained Knowledge Unlearning
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));University of Macau, China(澳门大学);Nankai University, China(南开大学);Zhejiang University, China(浙江大学);Peng Cheng Laboratory, China(鹏城实验室);Shaqra University, Saudi Arabia(沙克拉大学)
MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Tongji University(同济大学)
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
Center for AI Safety(AI安全中心);University of California, Berkeley(加州大学伯克利分校);Massachusetts Institute of Technology(麻省理工学院);SecureBio;Scale AI;New York University(纽约大学);IIIT Hyderabad(海德拉巴印度信息技术学院);Stanford University(斯坦福大学);Harvard University(哈佛大学);University of Southern California(南加州大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);University of California, Los Angeles(加州大学洛杉矶分校);California Institute of Technology(加州理工学院);Sybil;Northeastern University(东北大学)