擦除幻觉：对LLM遗忘评估泛化能力的压测试
The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation
AI总结：
本文提出PSG框架，通过生成语义衍生但嵌入空间不同的替代数据集，揭示LLM无学习评估中普遍存在的度量不准确问题。
摘要：
机器无学习旨在从训练模型中移除特定数据的影响，这种能力对于遵守版权法和确保人工智能安全至关重要。当前的无学习度量通常通过监控模型在特定无学习数据集（D_u）上的性能下降来衡量成功。我们认为，对于大型语言模型（LLMs），这种评估范式是不足且可能误导的。许多现实世界中的无学习应用——受版权或安全动机驱动——不仅隐式地针对D_u中的逐字内容，还针对模型由此推导出的更广泛泛化行为所影响的行为。我们证明LLMs可以通过标准无学习评估并似乎“忘记”了目标知识，同时在与D_u语义相邻的内容上保留了强大的能力。这种现象表明，擦除确切句子并不一定等同于移除底层知识。为了解决这一差距，我们提出了近似替代生成（PSG），一个自动压测试框架，生成一个替代数据集，D_u。这个替代集是基于D_u语义推导的，但在嵌入空间中足够不同。通过比较D_u和D_u之间的无学习度量分数，我们可以对度量本身进行压力测试。我们对三种LLM家族（Llama-3-8B、Qwen2.5-7B和Zephyr-7B-β）、三个不同的数据集和七个标准度量进行了广泛的评估，发现广泛的一致性问题。我们发现当前度量经常高估无学习的成功率，未能检测到我们压力测试数据集暴露的保留知识。
英文摘要：
Machine unlearning aims to remove specific data influences from trained models, a capability essential for adhering to copyright laws and ensuring AI safety. Current unlearning metrics typically measure success by monitoring the model's performance degradation on the specific unlearning dataset ($D_u$). We argue that for Large Language Models (LLMs), this evaluation paradigm is insufficient and potentially misleading. Many real-world uses of unlearning--motivated by copyright or safety--implicitly target not only verbatim content in $D_u$, but also behaviors influenced by the broader generalizations the model derived from it. We demonstrate that LLMs can pass standard unlearning evaluation and appear to have "forgotten" the target knowledge, while simultaneously retaining strong capabilities on content that is semantically adjacent to $D_u$. This phenomenon indicates that erasing exact sentences does not necessarily equate to removing the underlying knowledge. To address this gap, we propose Proximal Surrogate Generation (PSG), an automated stress-testing framework that generates a surrogate dataset, $\tilde{D}_u$. This surrogate set is constructed to be semantically derived from $D_u$ yet sufficiently distinct in embedding space. By comparing unlearning metric scores between $D_u$ and $\tilde{D}_u$, we can stress-test the reliability of the metric itself. Our extensive evaluation across three LLM families (Llama-3-8B, Qwen2.5-7B, and Zephyr-7B-$β$), three distinct datasets, and seven standard metrics reveals widespread inconsistencies. We find that current metrics frequently overestimate unlearning success, failing to detect retained knowledge exposed by our stress-test datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
