知识蒸馏（Knowledge Distillation, KD）将大型教师模型的知识迁移到更小的学生模型中。尽管对比目标已被证明在自监督场景下学习结构化表征有效，但它在蒸馏中的应用受限于两个实际缺陷：依赖外部记忆库进行负采样，以及固定的温度超参数限制了在不同训练阶段和师生模型对之间的适应性。因此我们提出判别性与一致性表征蒸馏（Discriminative and Consistent Representation Distillation, DCD），该方法将对比实例判别与跨模型相似度矩阵上的一致性正则项相结合。对比项将每个学生表征与对应的教师表征对齐，而一致性项对该矩阵行归一化视图和列归一化视图之间的不对称性进行惩罚，约束仅靠实例判别无法约束的非对角结构；我们证明当该矩阵为对称矩阵时，一致性项恰好为零。我们进一步引入了高效的批内采样方法，无需外部记忆库，并设置可学习的缩放和偏置参数，在训练过程中自适应控制蒸馏信号的锐度和偏移量。该方法与标准KD的训练速度相当，仅额外增加6.6万个参数。通过在CIFAR-100、ImageNet和MS-COCO上的大量实验，以及向STL-10和Tiny ImageNet的跨数据集迁移实验，我们表明该方法在分类、目标检测和迁移任务中取得了具有竞争力的性能，同时与现有对比蒸馏方法相比，大幅降低了内存消耗和训练时间。
英文摘要：
Knowledge Distillation (KD) transfers knowledge from a large teacher to a smaller student model. While contrastive objectives have proven effective for learning structured representations in self-supervised settings, their use in distillation is hindered by two practical shortcomings: the reliance on external memory banks for negative sampling, and fixed temperature hyperparameters that limit adaptability across training stages and teacher-student pairs. We therefore propose Discriminative and Consistent Representation Distillation (DCD), which combines contrastive instance discrimination with a consistency regularization term over the cross-model similarity matrix. The contrastive term aligns each student representation with its teacher counterpart, while the consistency term penalizes asymmetry between the row-normalized and column-normalized views of that matrix, constraining the off-diagonal structure that instance discrimination alone leaves free; we show that it vanishes precisely when this matrix is symmetric. We further introduce an efficient in-batch sampling that eliminates external memory banks, and learnable scale and bias parameters that adapt during training to control the sharpness and offset of the distillation signal. The method matches the training speed of standard KD while adding only 66K additional parameters. Through extensive experiments on CIFAR-100, ImageNet, and MS-COCO, together with cross-dataset transfer to STL-10 and Tiny ImageNet, we show that our approach achieves competitive performance in classification, object detection, and transfer, while substantially reducing memory consumption and training time compared to existing contrastive distillation methods.
DeCO: Discriminative Evidence Composition for Fine-Grained Dataset Distillation
University of Science and Technology of China（中国科学技术大学）；Hokkaido University（北海道大学）；The University of Queensland（昆士兰大学）；Shanghai AI Laboratory（上海人工智能实验室）；Dalian University of Technology（大连理工大学）
Sun Yat-sen University（中山大学）；Key Laboratory of Machine Intelligence and Advanced Computing, MOE（机器智能与先进计算教育部重点实验室）
Successful continual learning of new knowledge would enable intelligent systems to recognize more and more classes of objects. However, current intelligent systems often fail to correctly recognize previously learned classes of objects when updated to learn new classes. It is widely believed that such downgraded performance is solely due to the catastrophic forgetting of previously learned knowledge. In this study, we argue that the class confusion phenomena may also play a role in downgrading the classification performance during continual learning, i.e., the high similarity between new class…
Toward Realistic Remote Sensing Dataset Distillation with Discriminative Prototype-guided Diffusion
Computer Vision and Learning Systems, Department of Electrical Engineering, Linköping University（计算机视觉与学习系统，电气工程系，利厄普大学）；Helmholtz-Zentrum Dresden-Rossendorf, Responsible AI Group（海姆霍尔茨·德累斯顿-罗斯托克研究中心，负责任人工智能小组）；University of Iceland, Faculty of Electrical and Computer Engineering（冰岛大学，电气与计算机工程学院）
FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation
Zhejiang University（浙江大学）；Hokkaido University（北海道大学）；The University of Queensland（昆士兰大学）；Shanghai AI Laboratory（上海人工智能实验室）；Dalian University of Technology（大连理工大学）
Istituto Italiano di Tecnologia（意大利技术研究院）；University of Genova（热那亚大学）；University of Verona（维罗纳大学）
Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a …
Data-free Knowledge Distillation for Fine-grained Visual Categorization
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）
The performance of a distillation-based compressed network is governed by the quality of distillation. The reason for the suboptimal distillation of a large network (teacher) to a smaller network (student) is largely attributed to the gap in the learning capacities of given teacher-student pair. While it is hard to distill all the knowledge of a teacher, the quality of distillation can be controlled to a large extent to achieve better performance. Our experiments show that the quality of distillation is largely governed by the quality of teacher's response, which in turn is heavily affected by…
Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Plasma Physics, Chinese Academy of Sciences（中国科学院等离子体物理研究所）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University（安徽大学光电子信息获取与控制技术国家重点实验室）
Point cloud classification is a popular task in 3D vision. However, previous works, usually assume that point clouds at test time are obtained with the same procedure or sensor as those at training time. Unsupervised Domain Adaptation (UDA) instead, breaks this assumption and tries to solve the task on an unlabeled target domain, leveraging only on a supervised source domain. For point cloud classification, recent UDA methods try to align features across domains via auxiliary tasks such as point cloud reconstruction, which however do not optimize the discriminative power in the target domain i…
Generalization Performance of Deep Learning models trained using Empirical Risk Minimization can be improved significantly by using Data Augmentation strategies such as simple transformations, or using Mixed Samples. We attempt to empirically analyze the impact of such strategies on the transfer of generalization between teacher and student models in a distillation setup. We observe that if a teacher is trained using any of the mixed sample augmentation strategies, such as MixUp or CutMix, the student model distilled from it is impaired in its generalization capabilities. We hypothesize that s…
The University of Queensland（昆士兰大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61部门）
In knowledge distillation, previous feature distillation methods mainly focus on the design of loss functions and the selection of the distilled layers, while the effect of the feature projector between the student and the teacher remains under-explored. In this paper, we first discuss a plausible mechanism of the projector with empirical evidence and then propose a new feature distillation method based on a projector ensemble for further performance improvement. We observe that the student network benefits from a projector even if the feature dimensions of the student and the teacher are the …
Chongqing University（重庆大学）；San Diego State University（圣地亚哥州立大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Multi-Label Image Classification (MLIC) approaches usually exploit label correlations to achieve good performance. However, emphasizing correlation like co-occurrence may overlook discriminative features of the target itself and lead to model overfitting, thus undermining the performance. In this study, we propose a generic framework named Parallel Self-Distillation (PSD) for boosting MLIC models. PSD decomposes the original MLIC task into several simpler MLIC sub-tasks via two elaborated complementary task decomposition strategies named Co-occurrence Graph Partition (CGP) and Dis-occurrence G…
Despite the success that metric learning based approaches have achieved in few-shot learning, recent works reveal the ineffectiveness of their episodic training mode. In this paper, we point out two potential reasons for this problem: 1) the random episodic labels can only provide limited supervision information, while the relatedness information between the query and support samples is not fully exploited; 2) the meta-learner is usually constrained by the limited contextual information of the local episode. To overcome these problems, we propose a new Global Relatedness Decoupled-Distillation…
AutoAugment has been a powerful algorithm that improves the accuracy of many vision tasks, yet it is sensitive to the operator space as well as hyper-parameters, and an improper setting may degenerate network optimization. This paper delves deep into the working mechanism, and reveals that AutoAugment may remove part of discriminative information from the training image and so insisting on the ground-truth label is no longer the best option. To relieve the inaccuracy of supervision, we make use of knowledge distillation that refers to the output of a teacher model to guide network training. Ex…
Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Xiamen Data Intelligence Academy of ICT, CAS（中国科学院计算技术研究所厦门数据智能研究院）
Previous Online Knowledge Distillation (OKD) often carries out mutually exchanging probability distributions, but neglects the useful representational knowledge. We therefore propose Multi-view Contrastive Learning (MCL) for OKD to implicitly capture correlations of feature embeddings encoded by multiple peer networks, which provide various views for understanding the input data instances. Benefiting from MCL, we can learn a more discriminative representation space for classification than previous OKD methods. Experimental results on image classification demonstrate that our MCL-OKD outperform…
In video understanding, most cross-modal knowledge distillation (KD) methods are tailored for classification tasks, focusing on the discriminative representation of the trimmed videos. However, action detection requires not only categorizing actions, but also localizing them in untrimmed videos. Therefore, transferring knowledge pertaining to temporal relations is critical for this task which is missing in the previous cross-modal KD frameworks. To this end, we aim at learning an augmented RGB representation for action detection, taking advantage of additional modalities at training time throu…
Despite significant advancements of deep learning-based forgery detectors for distinguishing manipulated deepfake images, most detection approaches suffer from moderate to significant performance degradation with low-quality compressed deepfake images. Because of the limited information in low-quality images, detecting low-quality deepfake remains an important challenge. In this work, we apply frequency domain learning and optimal transport theory in knowledge distillation (KD) to specifically improve the detection of low-quality compressed deepfake images. We explore transfer learning capabil…
University of Chinese Academy of Sciences（中国科学院大学）；Megvii Inc.（旷视科技有限公司）；Megvii Technology（旷视科技）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Institute of Basic Medical Sciences（北京基础医学研究所）
Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained at once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea …
The backbone of traditional CNN classifier is generally considered as a feature extractor, followed by a linear layer which performs the classification. We propose a novel loss function, termed as CAM-loss, to constrain the embedded feature maps with the class activation maps (CAMs) which indicate the spatially discriminative regions of an image for particular categories. CAM-loss drives the backbone to express the features of target category and suppress the features of non-target categories or background, so as to obtain more discriminative feature representations. It can be simply applied i…
Incomplete Multi-View Multi-Label Classification via Shared Codebook and Fused-Teacher Self-Distillation
College of Information Engineering, Shanghai Maritime University（上海海事大学信息工程学院）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Washington（华盛顿大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Medical Optical Technology R&D Center, Research Institute of Tsinghua（清华研究院医学光学技术研发中心）；Jinfeng Laboratory（金凤实验室）
YITU Technology（依图科技）；Northeastern University of China（中国东北大学）
Knowledge Distillation (KD) refers to transferring knowledge from a large model to a smaller one, which is widely used to enhance model performance in machine learning. It tries to align embedding spaces generated from the teacher and the student model (i.e. to make images corresponding to the same semantics share the same embedding across different models). In this work, we focus on its application in face recognition. We observe that existing knowledge distillation models optimize the proxy tasks that force the student to mimic the teacher's behavior, instead of directly optimizing the face …
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海高等研究院）；ZJU-Bangsun Joint Research Center（浙江大学-邦盛联合研究中心）；Zhejiang University of Technology（浙江工业大学）
Knowledge distillation aims to compress a powerful yet cumbersome teacher model into a lightweight student model without much sacrifice of performance. For this purpose, various approaches have been proposed over the past few years, generally with elaborately designed knowledge representations, which in turn increase the difficulty of model development and interpretation. In contrast, we empirically show that a simple knowledge distillation technique is enough to significantly narrow down the teacher-student performance gap. We directly reuse the discriminative classifier from the pre-trained …
Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam（河内科技大学电气与电子工程学院，越南）；Signal Analysis and Interpretation Lab, University of Southern California, United States（南加州大学信号分析与解释实验室，美国）
Unsupervised representation learning has proved to be a critical component of anomaly detection/localization in images. The challenges to learn such a representation are two-fold. Firstly, the sample size is not often large enough to learn a rich generalizable representation through conventional techniques. Secondly, while only normal samples are available at training, the learned features should be discriminative of normal and anomalous samples. Here, we propose to use the "distillation" of features at various layers of an expert network, pre-trained on ImageNet, into a simpler cloner network…
There is a growing discrepancy in computer vision between large-scale models that achieve state-of-the-art performance and models that are affordable in practical applications. In this paper we address this issue and significantly bridge the gap between these two types of models. Throughout our empirical investigation we do not aim to necessarily propose a new method, but strive to identify a robust and effective recipe for making state-of-the-art large scale models affordable in practice. We demonstrate that, when performed correctly, knowledge distillation can be a powerful tool for reducing…
Shanghai Jiao Tong University（上海交通大学）；École Centrale de Lyon（里昂中央理工学院）
Domain adaptation (DA) is transfer learning which aims to learn an effective predictor on target data from source data despite data distribution mismatch between source and target. We present in this paper a novel unsupervised DA method for cross-domain visual recognition which simultaneously optimizes the three terms of a theoretically established error bound. Specifically, the proposed DA method iteratively searches a latent shared feature subspace where not only the divergence of data distributions between the source domain and the target domain is decreased as most state-of-the-art DA meth…
The knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model (a.k.a. the "student"), which expands the student's knowledge and improves its learning efficacy. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space -- in this "Generalized Knowledge Distillation (GKD)", the classes of the teacher and the student may be the same, completely different, or …
Nanjing University of Science and Technology（南京理工大学）
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decouple…
Dataset Distillation for Histopathology Image Classification
Australian Institute of Health Innovation (AIHI), Macquarie University（澳大利亚麦考瑞大学澳大利亚健康创新研究所）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, Peking University（北京大学计算机学院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
