Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly. However, redundancy still exists in the learned representations through these prevalent methods, which tend to learn each spatial location's features indiscriminately. To derive a more compact representation (concept feature) from the teacher, inspired by human cognition, we suggest an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The source code is available at https://github.com/ZhgLiu/GDD.
Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);Institute of High-Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Elucidating the Preconditioning in Consistency Distillation
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab(计算机科学与技术系,人工智能研究所,BNRist中心,THBI实验室);Tsinghua-Bosch Joint ML Center, Tsinghua University(清华大学-博世联合机器学习中心,清华大学);Shengshu Technology(盛舒科技);Pazhou Lab (Huangpu), Guangzhou, China(黄埔实验室(广州),中国)
Multiple Degradation and Reconstruction Network for Single Image Denoising via Knowledge Distillation
The Chinese University of Hong Kong(香港中文大学);East China Normal University(华东师范大学);Nanjing University of Posts and Telecommunications(南京邮电大学)
随着深度学习的发展,单幅图像去噪(SID)取得了重大突破。然而,所提出的方法往往伴随着大量的参数,这极大地限制了它们的应用场景。与以往盲目增加网络深度的方法不同,本文研究了含噪图像的退化机制,提出了一种轻量级的多重退化与重构网络(Multiple Degradation and Reconstruction Network,MDRN)来逐步去除噪声。同时,我们提出了两种新的异构知识蒸馏策略(HMDS),使MDRN学习更丰富,更准确的特征,从异构模型,这使得它有可能在极端条件下重建更高质量的去噪图像。大量的实验表明,我们的MDRN实现了良好的性能对其他SID模型与较少的参数。同时,大量的烧蚀研究表明,引入HMDS可以改善微小模型或高噪声水平下模型的性能,这在相关应用中是非常有用的。
Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
Digital Environment Research Institute (DERI), Queen Mary University of London(女王玛丽大学数字环境研究所);School of Electronic Engineering and Computer Science, Queen Mary University of London(女王玛丽大学电子工程与计算机科学学院)
GaitKD: A Universal Decoupled Distillation Framework for Efficient Gait Recognition
The City University of New York(纽约城市大学);Wuhan University(武汉大学);Beijing Jiaotong University(北京交通大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Oulu(奥卢大学)