迈向抗蒸馏的大语言模型：信息论视角
Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective
AI总结：
本文从信息论角度分析蒸馏防御问题，提出通过最小化条件互信息来提升模型抗蒸馏能力，设计变换矩阵净化输出以增强安全性，实验表明方法有效降低蒸馏性能并保持任务准确性。
摘要：
专有大语言模型（LLMs）具有显著经济价值，通常仅以黑盒API形式暴露，但攻击者仍可通过蒸馏提取知识。现有防御仅关注基于文本的蒸馏，未探索基于logit的蒸馏。本文从信息论视角分析该问题，通过计算教师输出中logit与输入查询在真实标签条件下的条件互信息（CMI），提出通过最小化CMI来防御蒸馏。基于理论分析，我们提出学习变换矩阵以净化原始输出，增强抗蒸馏能力。进一步推导CMI启发的反蒸馏目标以优化此变换，有效去除蒸馏相关信息同时保持输出效用。在多个LLM和强蒸馏算法上的广泛实验表明，所提方法显著降低蒸馏性能，同时保持任务准确性，有效保护模型知识产权。
英文摘要：
Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
