arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01735cs.CL
收藏

少即是多:面向LLM的几何遗忘方法,实现最小数据披露

Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure

Faculty of Information Technology, Monash University, Clayton, Victoria, Australia.(墨尔本大学信息技术学院,澳大利亚维多利亚州克莱顿分校)

Chenchen Tan, Xinghao Li, Shujie Cui, Youyang Qu, Cunjian Chen, Longxiang Gao

AI总结:

提出几何遗忘(GU)方法,通过操作模型隐藏状态并利用少量安全参考提示和锚点上下文合成提示,在无需原始训练数据的情况下实现高效选择性遗忘,在ToFU和UnlearnPII基准上达到强目标抑制且对非目标性能影响最小。

摘要:

随着大型语言模型(LLM)越来越多地部署在现实系统中,它们必须支持事后移除特定内容以满足隐私和治理要求。这推动了选择性遗忘的发展,即抑制关于特定实体或主题的信息,同时保持LLM的通用效用。然而,现有的大多数LLM遗忘方法需要访问原始训练语料库,并依赖于输出级拒绝微调或广泛的梯度更新,在遗忘强度、非目标保持和数据可用性之间造成紧张。我们提出几何遗忘(GU),一种直接操作模型提示条件隐藏状态的方法,无需访问原始训练语料库。具体来说,GU从少量安全参考提示中提炼出紧凑的低秩安全行为子空间,并使用轻量级的锚点上下文合成提示触发隐藏表示向该安全子空间的局部投影对齐。对合成非目标锚点的教师蒸馏正则化进一步减少了附带漂移。在面向隐私的遗忘基准(ToFU和UnlearnPII)上,GU实现了强目标抑制,对非目标性能影响最小,表明有效遗忘可以用最少的合成数据实现。

英文摘要:

As large language models (LLMs) are increasingly deployed in real-world systems, they must support post-hoc removal of specific content to meet privacy and governance requirements. This motivates selective unlearning, which suppresses information about a particular entity or topic while preserving the LLM's general utility. However, most existing LLM unlearning methods require access to the original training corpus and rely on output-level refusal tuning or broad gradient updates, creating a tension among unlearning strength, non-target preservation, and data availability. We propose Geometric Unlearning (GU), an approach that operates directly on the model's prompt-conditioned hidden states without access to the original training corpus. Specifically, GU distills a compact, low-rank safe-behavior subspace from a small set of safe reference prompts and uses lightweight anchor-in-context synthetic prompts to trigger localized, projection-based alignment of hidden representations to this safe subspace. A teacher-distillation regularizer on synthetic non-target anchors further reduces collateral drift. Across privacy-oriented unlearning benchmarks (ToFU and UnlearnPII), GU achieves strong target suppression with minimal impact on non-target performance, demonstrating that effective unlearning can be achieved with minimal synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑