arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11266cs.AI
收藏

消除幻觉:一种动态框架用于评估大语言模型的消除

The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning

Georgia Institute of Technology(佐治亚理工学院);Stanford University(斯坦福大学);IBM Research(IBM研究院)

Raj Sanjay Shah, Jing Huang, Keerthiram Murugesan, Nathalie Baracaldo, Diyi Yang

AI总结:

本文提出了一种动态框架,用于评估大语言模型消除方法的鲁棒性,通过复杂结构查询揭示消除技术在多跳设置中的脆弱性,并提供可扩展的评估方法。

摘要:

在大语言模型(LLM)中,消除旨在提高安全性、减轻偏见并遵守法律要求,如被遗忘权。然而,现有的消除方法是脆弱的:微小的查询修改,如多跳推理和实体别名,可以恢复 supposedly 忘记的信息。因此,当前的评估指标往往产生有效性幻觉,由于依赖静态、无结构的基准,无法检测这些漏洞。我们提出了一种动态框架,通过复杂的结构查询来压力测试消除鲁棒性。我们的方法首先从目标模型(预消除)中提取知识,并构建目标探测器,从简单的查询到多跳链,允许对查询难度进行精确控制。我们的实验表明,该框架(1)通过自动生成语义等价的问答探测器,显示出与现有基准相当的覆盖范围;(2)与先前评估一致;(3)揭示了其他基准未发现的消除失败,特别是在多跳设置中。此外,激活分析显示,单跳查询通常遵循主导的计算路径,这些路径更可能被消除方法破坏。相比之下,多跳查询倾向于使用替代路径,通常保持完整,解释了消除技术在多跳设置中的脆弱性。我们的框架使消除方法的评估变得实用且可扩展,而无需手动构建遗忘测试集,从而更容易在实际应用中采用。我们发布了pip包和代码,网址为https://sites.google.com/view/unlearningmirage/home。

英文摘要:

Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly forgotten information. As a result, current evaluation metrics often create an illusion of effectiveness, failing to detect these vulnerabilities due to reliance on static, unstructured benchmarks. We propose a dynamic framework that stress tests unlearning robustness using complex structured queries. Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains, allowing precise control over query difficulty. Our experiments show that the framework (1) shows comparable coverage to existing benchmarks by automatically generating semantically equivalent Q&A probes, (2) aligns with prior evaluations, and (3) uncovers new unlearning failures missed by other benchmarks, particularly in multi-hop settings. Furthermore, activation analyses show that single-hop queries typically follow dominant computation pathways, which are more likely to be disrupted by unlearning methods. In contrast, multi-hop queries tend to use alternative pathways that often remain intact, explaining the brittleness of unlearning techniques in multi-hop settings. Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets, enabling easier adoption for real-world applications. We release the pip package and the code at https://sites.google.com/view/unlearningmirage/home.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑