蒸馏博弈：自适应攻击与高效防御
The Distillation Game: Adaptive Evaluations & Efficient Defenses
- Stanford University（斯坦福大学）
- Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
- National University of Singapore（新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
通过最小化博弈框架研究蒸馏攻击中模型提供者的部署权衡，提出自适应评估规则和产品专家（PoE）防御方法，实验表明自适应学生能恢复更多能力，且PoE在成本和质量上具有优势。
AI中文摘要：
蒸馏攻击为模型提供者带来了部署权衡：使模型更有用的相同输出也可能使其更容易被模仿。我们通过一个效用受限的教师和自适应学生之间的最小化博弈来研究这种权衡。我们的框架产生了可处理的一侧响应规则：一个自适应评估规则，其中学生重新加权高价值示例，以及一个教师侧防御模板，抑制对蒸馏最有用的输出。从示例价值的廉价代理中，我们推导出产品专家（PoE），一种简单的前向传递防御，在生成过程中将教师与代理学生结合。实验上，自适应评估揭示了一个大的被动-自适应差距：在最先进的防御上，自适应学生在GSM8K和MATH上恢复了比被动评估所建议的更多的能力。在这种更强的评估下，昂贵防御和PoE之间的明显鲁棒性差距显著缩小，而PoE仍然便宜得多，并保留了更高质量的推理轨迹。总体而言，我们的结果表明，强大的蒸馏仍然难以阻止，并且反蒸馏的进展应该根据自适应学生而非被动学生来判断。我们的代码可在：https://github.com/ysfalh/distillation-game 获取。
英文摘要：
Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitate. We study this trade-off through a minimax game between a utility-constrained teacher and an adaptive student. Our framework yields tractable one-sided response rules: an adaptive evaluation rule in which the student reweights high-value examples, and a teacher-side defense template that suppresses outputs most useful for distillation. From a cheap proxy for example value, we derive Product-of-Experts (PoE), a simple forward-pass-only defense that combines the teacher with a proxy student during generation. Empirically, adaptive evaluation reveals a large passive--adaptive gap: on state-of-the-art defenses, adaptive students recover substantially more capability than passive evaluation suggests on GSM8K and MATH. Under this stronger evaluation, the apparent robustness gap between expensive defenses and PoE narrows considerably, while PoE remains substantially cheaper and preserves higher-quality reasoning traces. Overall, our results suggest that strong distillation remains difficult to stop, and that progress on antidistillation should be judged against adaptive students rather than passive ones. Our code is available at: https://github.com/ysfalh/distillation-game.
