单教师视图增强：通过学生引导的扰动增强知识蒸馏

中文摘要 AI 辅助

知识蒸馏通常依赖单一教师的固定视角，限制了监督信号的多样性。多教师蒸馏虽能解决此问题，但计算和存储成本过高。为平衡效率与多样性，近期研究聚焦于从单一教师生成虚拟视图。现有方法存在权衡：随机扰动方法高效但缺乏可控多样性，结构化增强方法需多阶段训练且参数线性增长。我们提出Shift-Augmented Knowledge Distillation（SAKD）框架，利用学生不断演变的特征作为扰动生成的动态条件，实现单阶段训练并通过无参数循环移位产生自适应、多样的视图。在CIFAR-100和ImageNet上的大量实验表明，SAKD始终优于随机扰动方法，且在参数显著减少并消除预训练要求的情况下，达到与两阶段方法相当的准确率。

英文摘要

Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher's strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation. This shift in perspective enables single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements.

发表机构

• State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能技术国家重点实验室）
• Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）
• Institute of Collaborative Innovation, University of Macau（澳门大学协同创新研究院）
• Beijing Huahang Institute of Radio Measurement（北京华航无线电测量研究所）

机构由 AI 辅助整理，请以论文原文为准。