arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.01699cs.LGcs.AIcs.CRcs.NE
收藏

探测器几何对齐:在偶然机会以下消除跨序列记忆签名

Probe-Geometry Alignment: Erasing the Cross-Sequence Memorization Signature Below Chance

Howard University(霍华德大学)

Anamika Paul Rupa, Anietie Andy

AI总结:

本文研究了大语言模型中记忆痕迹的消除方法,提出通过探测器几何对齐技术在不损害能力的前提下,有效消除跨序列记忆签名,且在多种模型上验证了其有效性。

摘要:

最近的攻击表明,大型语言模型的行为遗忘会留下可被对抗探测器恢复的内部痕迹。我们刻画了这种保留的位置,并展示其可以被手术性地移除而不产生可测量的能力成本。我们的核心协议是一种留一法跨序列探测器,用于测试记忆签名是否能跨被排除的序列泛化。该签名在不同规模上都是真实且一致的:在Pythia-70M、GPT-2中等和Mistral-7B上,记忆特定的缺口分别为+0.32、+0.19和+0.30;在Pythia-70M上,随机初始化的对照在最深的层中坍缩到-0.04,这是预训练签名峰值的最深层。探测方向与回忆是因果分离的——投影它出去会局部消除签名(+0.44 -> -0.19),而行为回忆几乎没有变化——并且一个在自然记忆内容上训练的探测器不会分类微调注入的秘密,这标志着两个表征上不同的领域。我们随后引入探测器几何对齐(PGA),一种手术性消除,使在每个深度沿探测器的活读出方向对齐激活。PGA在所有测试的四个尺度上将跨序列探测器降至随机机会以下(玩具深度-4:0.17;Pythia-70M:0.07;Mistral-7B:0.45;GPT-2中等:0.06 via MD-PGA k=2),并在六种对抗探测器变体中保持稳健。针对重新训练攻击者,我们扩展PGA,使其在所有记忆相关深度上击败重新训练的探测器,同时在每个任务内保持五个零样本能力基准在2.8个百分点以内(平均Δacc = +0.2pp)。跨序列签名是预训练表示的一个真实、因果分离、领域特定的属性——通过每次深度的单个秩一干预可被在不产生可测量的能力成本下消除。

英文摘要:

Recent attacks show that behavioural unlearning of large language models leaves internal traces recoverable by adversarial probes. We characterise where this retention lives and show it can be surgically removed without measurable capability cost. Our central protocol is a leave-one-out cross-sequence probe that tests whether a memorisation signature generalises across held-out sequences. The signature is real and consistent across scale: memorisation-specific gaps of +0.32, +0.19, +0.30 on Pythia-70M, GPT-2 medium, and Mistral-7B; on Pythia-70M, the random-initialisation control collapses to -0.04 at the deepest layer where the pretrained signature peaks. The probe direction is causally separable from recall -- projecting it out collapses the signature locally (+0.44 -> -0.19) while behavioural recall barely changes -- and a probe trained on naturally memorised content does not classify fine-tuning-injected secrets, marking two representationally distinct regimes. We then introduce probe-geometry alignment (PGA), a surgical erasure that aligns activations along the probe's live readout direction at each depth. PGA drives the cross-sequence probe below random chance at all four scales tested (toy depth-4: 0.17; Pythia-70M: 0.07; Mistral-7B: 0.45; GPT-2 medium: 0.06 via MD-PGA k=2) and remains robust to six adversarial probe variants. Against a re-fitting attacker who trains a fresh probe on PGA-treated activations, we extend PGA adversarially, defeating the re-fit probe at every memorisation-relevant depth while preserving five zero-shot capability benchmarks within 2.8 percentage points per task (mean Δacc = +0.2pp). The cross-sequence signature is a real, causally separable, regime-specific property of pretrained representations -- removable below chance with a single rank-one intervention per depth at no measurable capability cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑