RWKU：面向大语言模型的现实世界知识遗忘基准测试
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
AI总结：
本文提出RWKU基准，在遗忘语料与保留语料均不可访问的现实设定下，以200位知名人物为遗忘目标，通过成员推断攻击和对抗探针等评估LLM知识遗忘效果。
摘要：
大语言模型（LLMs）不可避免地会从训练语料中记忆敏感、受版权保护以及有害的知识；因此，从模型中擦除这些知识至关重要。机器遗忘是一种通过事后修改模型来高效移除特定知识的有前景的解决方案。在本文中，我们提出了一个面向LLM遗忘的现实世界知识遗忘基准（RWKU）。RWKU基于以下三个关键因素设计：（1）在任务设置方面，我们考虑了一种更实际且更具挑战性的遗忘设置，即遗忘语料和保留语料均不可访问。（2）在知识来源方面，我们选取了200位现实世界知名人物作为遗忘目标，并表明这类流行知识广泛存在于各种LLM中。（3）在评估框架方面，我们设计了遗忘集和保留集，以评估模型在各类现实世界应用中的能力。针对遗忘集，我们提供了四种成员推断攻击（MIA）方法和九种对抗性攻击探针，以严格测试遗忘效果。针对保留集，我们从邻居扰动、通用能力、推理能力、真实性、事实性和流畅性等方面评估局部性和实用性。我们在两种遗忘场景、两个模型和六种基线方法上进行了大量实验，并获得了一些有意义的发现。我们将基准和代码公开发布在http://rwku-bench.github.io以供未来工作使用。
英文摘要：
Large language models (LLMs) inevitably memorize sensitive, copyrighted, and harmful knowledge from the training corpus; therefore, it is crucial to erase this knowledge from the models. Machine unlearning is a promising solution for efficiently removing specific knowledge by post hoc modifying models. In this paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM unlearning. RWKU is designed based on the following three key factors: (1) For the task setting, we consider a more practical and challenging unlearning setting, where neither the forget corpus nor the retain corpus is accessible. (2) For the knowledge source, we choose 200 real-world famous people as the unlearning targets and show that such popular knowledge is widely present in various LLMs. (3) For the evaluation framework, we design the forget set and the retain set to evaluate the model's capabilities across various real-world applications. Regarding the forget set, we provide four four membership inference attack (MIA) methods and nine kinds of adversarial attack probes to rigorously test unlearning efficacy. Regarding the retain set, we assess locality and utility in terms of neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. We conduct extensive experiments across two unlearning scenarios, two models and six baseline methods and obtain some meaningful findings. We release our benchmark and code publicly at http://rwku-bench.github.io for future work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
