短捷径胜长方法：为无数据知识蒸馏寻求多样且稳定的生成器
Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation
- Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
- Zhongguancun Academy（中关村学院）
- School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
- School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
- Shandong Computer Science Center（山东省计算机科学中心）
- Spatialtemporal AI（时空人工智能公司）
- Tongji University（同济大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对无数据知识蒸馏中生成器的频域捷径学习与训练不稳定问题，提出CSWL框架，通过特征级频域增强和CSFR辅助任务提升生成器多样性与稳定性，经多类图像数据集验证有效。
AI中文摘要：
无数据知识蒸馏（Data-Free Knowledge Distillation, DFKD）在无需访问真实数据的情况下传递知识，从而保护隐私。然而，现有的基于生成器的DFKD方法过度依赖教师偏好且存在模式崩溃问题，在频域中表现出“生成捷径学习”：依赖特定的频率分量和频率位置，导致合成图像质量和类别多样性不一致。本文提出CSWL框架，旨在从频域视角引入见解，以提高生成器的多样性和训练稳定性，从而“关闭捷径学习现象以在长期获胜”。为解决生成捷径学习问题，我们在特征级别引入频域增强，鼓励生成器关注全频谱，进而抑制捷径学习行为。为解决训练不稳定问题，我们提出跨阶段频率重建（Cross-Stage Frequency Reconstruction, CSFR）辅助任务，该任务隐式构建指数移动平均（Exponential Moving Average, EMA）机制，以促进长期优化和稳定性。大量实验（包括下游任务和多种分辨率的各类图像识别数据集）验证了CSWL从频域视角提升多样性和稳定性的有效性。
英文摘要：
Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.
