通过群体相对策略优化实现强化反学习
Reinforcement Unlearning via Group Relative Policy Optimization
AI总结：
本文提出PURGE方法，通过群体相对策略优化框架将反学习建模为可验证问题，有效减少敏感数据影响，提升模型流畅性和鲁棒性，实验表明其在反学习效果和实用性上优于现有方法。
摘要：
在预训练过程中，大语言模型（LLMs）无意中记忆了敏感或受版权保护的数据，这在GDPR和欧盟AI法案等法律框架下构成了重大合规挑战。满足这些要求需要能够从部署模型中删除信息而不重新训练的技术。现有的反学习方法试图解决这一需求，但往往泄露它们试图删除的数据，牺牲流畅性和鲁棒性，或依赖于昂贵的外部奖励模型。我们引入PURGE（通过相对群体擦除的策略反学习），一种基于群体相对策略优化框架的新方法，将反学习建模为可验证的问题。PURGE使用内在奖励信号，惩罚任何提及禁止概念的提及，从而实现安全且一致的反学习。我们的方法在目标上实现了比现有方法高达x46更低的每目标token使用量，同时在基模型上提高了5.48%的流畅性和12.02%的对抗鲁棒性。在现实世界知识反学习（RWKU）基准上的广泛评估表明，PURGE在保持98%原始效用的同时，达到了11%的反学习效果。PURGE表明，将LLM反学习建模为可验证任务能够实现更可靠、高效和可扩展的遗忘，表明了一种结合理论保证、改进安全性和实际部署效率的新反学习研究方向。
英文摘要：
During pretraining, LLMs inadvertently memorize sensitive or copyrighted data, posing significant compliance challenges under legal frameworks like the GDPR and the EU AI Act. Fulfilling these mandates demands techniques that can remove information from a deployed model without retraining from scratch. Existing unlearning approaches attempt to address this need, but often leak the very data they aim to erase, sacrifice fluency and robustness, or depend on costly external reward models. We introduce PURGE (Policy Unlearning through Relative Group Erasure), a novel method grounded in the Group Relative Policy Optimization framework that formulates unlearning as a verifiable problem. PURGE uses an intrinsic reward signal that penalizes any mention of forbidden concepts, allowing safe and consistent unlearning. Our approach achieves up to x46 lower token usage per target than state-of-the-art methods, while improving fluency by +5.48% and adversarial robustness by +12.02% over the base model. Extensive evaluation on the Real World Knowledge Unlearning (RWKU) benchmark shows that PURGE reaches 11% unlearning effectiveness while preserving 98% of original utility. PURGE shows that framing LLM unlearning as a verifiable task enables more reliable, efficient, and scalable forgetting, suggesting a promising new direction for unlearning research that combines theoretical guarantees, improved safety, and practical deployment efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
