Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly forgotten information. As a result, current evaluation metrics often create an illusion of effectiveness, failing to detect these vulnerabilities due to reliance on static, unstructured benchmarks. We propose a dynamic framework that stress tests unlearning robustness using complex structured queries. Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains, allowing precise control over query difficulty. Our experiments show that the framework (1) shows comparable coverage to existing benchmarks by automatically generating semantically equivalent Q&A probes, (2) aligns with prior evaluations, and (3) uncovers new unlearning failures missed by other benchmarks, particularly in multi-hop settings. Furthermore, activation analyses show that single-hop queries typically follow dominant computation pathways, which are more likely to be disrupted by unlearning methods. In contrast, multi-hop queries tend to use alternative pathways that often remain intact, explaining the brittleness of unlearning techniques in multi-hop settings. Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets, enabling easier adoption for real-world applications. We release the pip package and the code at https://sites.google.com/view/unlearningmirage/home.
On the importance of multiple training seeds for evaluating machine unlearning
Department of Computer Science(计算机科学系);School of Mathematics(数学学院);School of Science(科学学院);Statistics and Physics(统计学与物理学);Loughborough University(洛桑大学);Newcastle University(新castle大学)
MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Chinese University of Hong Kong(香港中文大学);Nanyang Technological University(南洋理工大学);University of Jinan(济南大学);Pengcheng Laboratory(鹏城实验室)
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Harbin Institute of Technology(哈尔滨工业大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));BIAI, ZJUT & D5Data.ai(BIAI、浙江工业大学及D5Data.ai);National University of Singapore(新加坡国立大学)
LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Technical University of Munich(慕尼黑技术大学);Siemens AG(西门子股份公司);University of Oxford(牛津大学)
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
Department of Computer Science(计算机科学系);University of North Carolina at Chapel Hill(北卡罗来纳大学 Chapel Hill 分校);School of Artificial Intelligence and Data Science(人工智能与数据科学学院);University of Science and Technology of China(中国科学技术大学)
SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation
Department of Computer Science, School of Science, Loughborough University(计算机科学系,科学学院,洛斯伯勒大学);School of Mathematics, Statistics and Physics, Newcastle University(数学、统计与物理学院,新卡克大学)
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
University of Maryland(马里兰大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);UK AI Security Institute(英国人工智能安全研究所);University of Toronto(多伦多大学);Northeastern University(东北大学);Haize Labs(Haize实验室);Dalhousie University(达尔豪斯大学);University of Waterloo(滑铁卢大学);University of Oxford(牛津大学);MIT(麻省理工学院)