读取微调先验：通过对比解码差异进行逐字内容恢复
Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing
AI总结：
提出对比解码差异（CDD）方法，仅基于输出层logit分布，无需权重或内部访问，即可从微调模型中逐字恢复植入事实，并在多种架构和场景下优于白盒方法。
摘要：
窄微调的语言模型会逐字记忆植入的内容，但在无法访问模型权重或训练数据的情况下，审计已部署模型所学内容仍然是一个开放挑战。最近的研究表明，基础模型与微调模型之间的激活差异携带了微调领域的可读痕迹；最先进的激活差异透镜（ADL）可以恢复模糊的领域级描述，但需要完全的“白盒”访问模型内部。我们引入了对比解码差异（CDD），一种仅操作输出级logit分布的模型差异方法，无需权重访问、无需层选择、无需针对模型调参，却能恢复植入的事实。CDD包含三个思想：绕过聊天模板以暴露原始微调先验，用最大模糊的前缀填充种子生成，以及在每个解码步骤放大微调模型与基础模型之间的logit空间差异。一个单一的默认配置即可逐字恢复植入的事实——精确的药物名称、投票数、物理测量值和程序细节——涵盖四种架构（1B-32B参数），尽管访问更少且运行速度约快170倍，但全面优于ADL。此外，CDD揭示了意外的数据管道伪影：LLM数据生成器通过模式崩溃引入的虚构角色泄露到模型权重中，并被CDD提取出来，据我们所知，这构成了首个从数据生成器伪影到模型权重再到恢复输出的端到端指纹链。我们在真实领域微调设置上进行了验证，在所有单数据集非CoT变体上实现了近乎完美的恢复，并在混合数据集设置中正确识别了所有四个数据集。CDD作为一种灰盒方法成功超越了白盒基线，凸显了其在AI系统透明度和问责性方面的实际效用。
英文摘要：
Narrowly finetuned language models memorize implanted content verbatim, but auditing what a deployed model has been taught, without access to its weights or training data, remains an open challenge. Recent work shows that activation differences between base and finetuned models carry readable traces of the finetuning domain; the state-of-the-art Activation Difference Lens (ADL) recovers a vague domain-level description but requires full "white-box" access to model internals. We introduce Contrastive Decoding Diffing (CDD), a model diffing method that operates on output-level logit distributions only, with no weight access, no layer selection, and no per-model tuning, yet recovers implanted facts. CDD consists of three ideas: bypassing the chat template to expose the raw finetuning prior, seeding generation with maximally vague pre-fills, and amplifying the logit-space difference between finetuned and base models at each decoding step. A single default configuration recovers implanted facts verbatim -- exact drug names, vote counts, physical measurements, and procedural details -- across four architectures (1B--32B parameters), uniformly outperforming ADL despite less access and running ~170x faster. Furthermore, CDD surfaces unintended data pipeline artifacts: a fictional persona introduced by the LLM data generator via mode collapse leaked into model weights and was extracted by CDD, constituting to our knowledge the first demonstrated end-to-end fingerprinting chain from data generator artifact to model weights to recovered output. We validate on real-domain finetuning settings, achieving near-perfect recovery across all single-dataset non-CoT variants and correctly identifying all four datasets in the mixed-dataset setting. CDD's success as a grey-box method outperforming white-box baselines underscores its practical utility for transparency and accountability in AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
