Data-free knowledge distillation~(DFKD) is an effective manner to solve model compression and transmission restrictions while retaining privacy protection, which has attracted extensive attention in recent years. Currently, the majority of existing methods utilize a generator to synthesize images to support the distillation. Although the current methods have achieved great success, there are still many issues to be explored. Firstly, the outstanding performance of supervised learning in deep learning drives us to explore a pseudo-supervised paradigm on DFKD. Secondly, current synthesized methods cannot distinguish the distributions of different categories of samples, thus producing ambiguous samples that may lead to an incorrect evaluation by the teacher. Besides, current methods cannot optimize the category-wise diversity samples, which will hinder the student model learning from diverse samples and further achieving better performance. In this paper, to address the above limitations, we propose a novel learning paradigm, i.e., conditional pseudo-supervised contrast for data-free knowledge distillation~(CPSC-DFKD). The primary innovations of CPSC-DFKD are: (1) introducing a conditional generative adversarial network to synthesize category-specific diverse images for pseudo-supervised learning, (2) improving the modules of the generator to distinguish the distributions of different categories, and (3) proposing pseudo-supervised contrastive learning based on teacher and student views to enhance diversity. Comprehensive experiments on three commonly-used datasets validate the performance lift of both the student and generator brought by CPSC-DFKD. The code is available at https://github.com/RoryShao/CPSC-DFKD.git
Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）
Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Waterloo（滑铁卢大学）
Synthetic Data Generation for Emotional Depth Faces: Optimizing Conditional DCGANs via Genetic Algorithms in the Latent Space and Stabilizing Training with Knowledge Distillation
Cyrus Intelligence Research Ltd（居鲁士智能研究有限公司）；Pars AI
Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack
State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Nanyang Technological University（南洋理工大学）
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, The University of Sydney（悉尼大学计算机学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
A Knowledge Distillation framework for Multi-Organ Segmentation of Medaka Fish in Tomographic Image
RPTU Kaiserslautern-Landau（莱茵兰-普法尔茨州凯泽斯劳滕-兰道应用技术大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg University（海德堡大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
Adaptive Dual-Teacher Distillation with Subnetwork Rectification for Bridging Semantic Gaps in Black-Box Domain Adaptation
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）
Beyond Classification: Knowledge Distillation using Multi-Object Impressions
Indian Institute of Science（印度科学理工学院）
知识蒸馏（KD）利用训练数据作为传递集将知识从复杂网络（教师）传递到较小网络（学生）。一些作品最近已经确定了许多情况下，训练数据可能是不可用的，由于数据隐私或敏感性的问题，并提出了解决方案，在此限制性约束的分类任务。与现有的作品不同，我们第一次解决了一个更具挑战性的问题，即，“KD for目标检测with zero knowledge about the训练数据及其统计”.我们提出的方法只使用预训练的Faster RCNN Teacher网络来准备伪目标并合成相应的样本（称为“多对象immersing”）。我们使用这个伪数据集作为一个转移集进行zero-shot KD的目标检测。我们证明了我们提出的方法的有效性，通过几个消融和广泛的实验基准数据集，如KITTI，Pascal和COCO。我们的方法没有训练样本，在具有相同和一半容量的学生上实现了64.2%和55.5%的可观mAP，同时在KITTI上从73.3% mAP的Resnet-18教师进行蒸馏。
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Toward Robust Semi-supervised Regression via Dual-stream Knowledge Distillation
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（重庆绿色智能技术研究所，中国科学院）；Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；Singapore Management University（新加坡管理大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Towards Generalized Multi-stage Clustering: Multi-view Self-distillation
Nankai University（南开大学）；Xingchuang Haihe Laboratory（兴创海河实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Stony Brook University（石溪大学）
DiffKD-DCIS: Predicting Upgrade of Ductal Carcinoma In Situ with Diffusion Augmentation and Knowledge Distillation
School of Medical Imaging, Laboratory Medicine and Rehabilitation, Xiangnan University（医学影像学院、实验室医学与康复学院，湘南大学）；Department of Radiotherapy, Affiliated Hospital (Clinical College) of Xiangnan University（放疗科，湘南大学附属医院（临床学院））；Faculty of Applied Sciences, Macao Polytechnic University（应用科学学院，澳门 polytechnic 大学）
Enhancing Automatic Chord Recognition via Pseudo-Labeling and Knowledge Distillation
California State University, Fullerton, CA, USA（加州州立大学弗里蒙特分校）；Microsoft, Redmond, WA, USA（微软公司）；Beijing Normal-Hong Kong Baptist University, China（北京师范大学-香港 Baptist大学）
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
诸如噪声条件得分网络和去噪扩散概率模型的迭代生成模型通过逐渐对初始噪声向量去噪来产生高质量样本。然而，它们的去噪过程有很多步骤，使它们比其他生成模型（如GANs和VAE）慢2-3个数量级。在本文中，我们建立了知识蒸馏和图像生成之间的一种新的连接，该技术将多步去噪过程蒸馏为单步的，从而获得与其他单步生成模型相似的采样速度。我们的去噪学生在CIFAR-10和CelebA数据集上生成与GANs相当的高质量样本，而无需对抗训练。我们证明了我们的方法通过在256 x 256 LSUN上的实验可以扩展到更高的分辨率。代码和检查点可在https://github.com/tcl9876/Denoising_Student上获得
Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation
Tsinghua University（清华大学）；Southern University of Science and Technology（南方科技大学）；The Hong Kong Polytechnic University（香港理工大学）；Southeast University（东南大学）
Knowledge Distillation Driven Semantic NOMA with GAN Refinement for 6G Robotic Vehicle Networks
School of Information and Electronics, Beijing Institute of Technology（北京理工大学信息与电子学院）；The Yangtze Delta Region Academy, Beijing Institute of Technology（北京理工大学长三角研究院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电机电子工程学系）；College of Computing, Birmingham City University（伯明翰城市大学计算机学院）
ARIA: Adaptive Region-Based Importance Allocation for Conditional Diffusion Distillation
Institute for AI, University of Stuttgart（斯图加特大学人工智能研究所）；IMPRS-IS（国际马克斯·普朗克智能系统研究学院）；Institute of Theoretical Chemistry, University of Stuttgart（斯图加特大学理论化学研究所）
