Transformer模型中的种子诱导唯一性:子空间对齐控制潜意识转移
Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer
AI总结:
研究揭示了Transformer模型中种子诱导的唯一性特性,通过子空间对齐影响潜意识转移,提出子空间级别的CKA诊断方法以提升多模型部署的安全性。
摘要:
我们分析了Transformer模型中的潜意识转移现象,其中教师嵌入了隐藏特征,这些特征可以被学生线性解码,而不会影响主要任务的表现。以往的研究通常将可转移性归因于全局表示相似性,通常用中心核对齐(CKA)来量化。使用具有解耦的公共和私人标签的合成语料库,我们在匹配和独立随机初始化下蒸馏学生。我们发现转移强度取决于一个特征判别子空间内的对齐程度:相同种子的学生继承这种对齐,并表现出更高的泄漏{τ≈}0.24,而不同种子的学生--尽管全局CKA>0.9--表现出显著减少的超额准确率{τ≈}0.12-0.13。我们通过子空间级别的CKA诊断和残差化探针正式化这一发现,显示泄漏跟踪特征判别子空间内的对齐程度,而不是全局表示相似性。安全控制(投影惩罚、对抗性反转、右因错误原因正则化)在不损害公共任务保真度的情况下减少相同基础模型的泄漏。这些结果确立了种子诱导的唯一性作为鲁棒性属性,并为安全多模型部署的子空间意识诊断提出论据。
英文摘要:
We analyze subliminal transfer in Transformer models, where a teacher embeds hidden traits that can be linearly decoded by a student without degrading main-task performance. Prior work often attributes transferability to global representational similarity, typically quantified with Centered Kernel Alignment (CKA). Using synthetic corpora with disentangled public and private labels, we distill students under matched and independent random initializations. We find that transfer strength hinges on alignment within a trait-discriminative subspace: same-seed students inherit this alignment and show higher leakage {τ\approx} 0.24, whereas different-seed students -- despite global CKA > 0.9 -- exhibit substantially reduced excess accuracy {τ\approx} 0.12 - 0.13. We formalize this with subspace-level CKA diagnostic and residualized probes, showing that leakage tracks alignment within the trait-discriminative subspace rather than global representational similarity. Security controls (projection penalty, adversarial reversal, right-for-the-wrong-reasons regularization) reduce leakage in same-base models without impairing public-task fidelity. These results establish seed-induced uniqueness as a resilience property and argue for subspace-aware diagnostics for secure multi-model deployments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。