Knowledge distillation between asymmetric architectures often induces severe geometric constraints on the learned representation space. We investigate dimensional collapse when distilling global Vision Transformers into capacity-constrained, local-receptive-field CNNs (0.5M-8.0M parameters). Using strictly centered SVD and Shannon Entropy Effective Rank, we confirm capacity-agnostic collapse under cosine distillation: a CLIP ViT-B/32 Teacher exhibits Effective Rank 88.68 on CIFAR-10, while all cosine-distilled students collapse to ~17 regardless of parameter count. An auxiliary InfoNCE objective expands this to ~41 dimensions. Critically, we ask whether this expansion is functionally useful. Multi-seed linear-probe evaluation shows InfoNCE expansion degrades downstream accuracy by 15-18 points relative to the collapsed baseline, despite more than doubling Effective Rank. A class-structure decomposition traces this to signal dilution: InfoNCE's class-blind uniformity pressure weakens class-discriminative structure in the original dimensions while adding only weakly relevant structure elsewhere. We then test a label-aware alternative, Supervised Contrastive distillation. On CIFAR-100, where we swept student capacity directly, its Effective Rank is invariant to capacity; on CIFAR-10, at a single tested width, it settles to a lower rank while matching baseline accuracy. Sweeping temperature instead of capacity, rank and downstream accuracy increase together monotonically. These results show Effective Rank alone is not a reliable proxy for representation quality: whether expansion helps or harms downstream performance depends on whether the driving objective is label-aware, not the magnitude of expansion itself.
This paper tackles the problem of semi-supervised video object segmentation on resource-constrained devices, such as mobile phones. We formulate this problem as a distillation task, whereby we demonstrate that small space-time-memory networks with finite memory can achieve competitive results with state of the art, but at a fraction of the computational cost (32 milliseconds per frame on a Samsung Galaxy S22). Specifically, we provide a theoretically grounded framework that unifies knowledge distillation with supervised contrastive representation learning. These models are able to jointly bene…
The University of Sydney（悉尼大学）；SenseTime Group Limited（商汤集团有限公司）
Knowledge distillation aims to transfer representation ability from a teacher model to a student model. Previous approaches focus on either individual representation distillation or inter-sample similarity preservation. While we argue that the inter-sample relation conveys abundant information and needs to be distilled in a more effective way. In this paper, we propose a novel knowledge distillation method, namely Complementary Relation Contrastive Distillation (CRCD), to transfer the structural knowledge from the teacher to the student. Specifically, we estimate the mutual relation in an anch…
Adversarial Robustness Distillation (ARD) is a novel method to boost the robustness of small models. Unlike general adversarial training, its robust knowledge transfer can be less easily restricted by the model capacity. However, the teacher model that provides the robustness of knowledge does not always make correct predictions, interfering with the student's robust performances. Besides, in the previous ARD methods, the robustness comes entirely from one-to-one imitation, ignoring the relationship between examples. To this end, we propose a novel structured ARD method called Contrastive Rela…
University of Rochester（罗切斯特大学）；JD AI Research（京东人工智能研究院）
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous anticipation tasks that aim at action label prediction, our work targets at generating natural language outputs that provide interpretable and accurate descriptions of future action steps. It is a challenging task due to the lack of semantic information extracted from the instructional videos. To overcome this challenge, we propose a novel knowledge distillation framework to exploit the related external textual knowledge …
While self-supervised representation learning (SSL) has proved to be effective in the large model, there is still a huge gap between the SSL and supervised method in the lightweight model when following the same solution. We delve into this problem and find that the lightweight model is prone to collapse in semantic space when simply performing instance-wise contrast. To address this issue, we propose a relation-wise contrastive paradigm with Relation Knowledge Distillation (ReKD). We introduce a heterogeneous teacher to explicitly mine the semantic information and transferring a novel relatio…
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；Xiamen University（厦门大学）；City University of Hong Kong（香港城市大学）
The amount of medical images for training deep classification models is typically very scarce, making these deep models prone to overfit the training data. Studies showed that knowledge distillation (KD), especially the mean-teacher framework which is more robust to perturbations, can help mitigate the over-fitting effect. However, directly transferring KD from computer vision to medical image classification yields inferior performance as medical images suffer from higher intra-class variance and class imbalance. To address these issues, we propose a novel Categorical Relation-preserving Contr…
The primary goal of knowledge distillation (KD) is to encapsulate the information of a model learned from a teacher network into a student network, with the latter being more compact than the former. Existing work, e.g., using Kullback-Leibler divergence for distillation, may fail to capture important structural knowledge in the teacher network and often lacks the ability for feature generalization, particularly in situations when teacher and student are built to address different classification tasks. We propose Wasserstein Contrastive Representation Distillation (WCoRD), which leverages both…
University of Toronto（多伦多大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
An effective framework for learning 3D representations for perception tasks is distilling rich self-supervised image features via contrastive learning. However, image-to point representation learning for autonomous driving datasets faces two main challenges: 1) the abundance of self-similarity, which results in the contrastive losses pushing away semantically similar point and image regions and thus disturbing the local semantic structure of the learned representations, and 2) severe class imbalance as pretraining gets dominated by over-represented classes. We propose to alleviate the self-sim…
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam（河内科技大学电气与电子工程学院，越南）；Signal Analysis and Interpretation Lab, University of Southern California, United States（南加州大学信号分析与解释实验室，美国）
It is a consensus that small models perform quite poorly under the paradigm of self-supervised contrastive learning. Existing methods usually adopt a large off-the-shelf model to transfer knowledge to the small one via distillation. Despite their effectiveness, distillation-based methods may not be suitable for some resource-restricted scenarios due to the huge computational expenses of deploying a large model. In this paper, we study the issue of training self-supervised small models without distillation signals. We first evaluate the representation spaces of the small models and make two non…
C2E: Boosting Ego-Only 3D Object Detection via Multi-Teacher Contrastive Knowledge Distillation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University（福建省城市智能感知与计算重点实验室，厦门大学）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算教育部重点实验室，厦门大学）；Zhongguancun Academy（中关村学院）
Sun Yat-sen University（中山大学）；SUNY at Buffalo（纽约州立大学布法罗分校）
Many unsupervised hashing methods are implicitly established on the idea of reconstructing the input data, which basically encourages the hashing codes to retain as much information of original data as possible. However, this requirement may force the models spending lots of their effort on reconstructing the unuseful background information, while ignoring to preserve the discriminative semantic information that is more important for the hashing task. To tackle this problem, inspired by the recent success of contrastive learning in learning continuous representations, we propose to adapt this …
Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Xiamen Data Intelligence Academy of ICT, CAS（中国科学院计算技术研究所厦门数据智能研究院）
Previous Online Knowledge Distillation (OKD) often carries out mutually exchanging probability distributions, but neglects the useful representational knowledge. We therefore propose Multi-view Contrastive Learning (MCL) for OKD to implicitly capture correlations of feature embeddings encoded by multiple peer networks, which provide various views for understanding the input data instances. Benefiting from MCL, we can learn a more discriminative representation space for classification than previous OKD methods. Experimental results on image classification demonstrate that our MCL-OKD outperform…
Vision-language models trained with contrastive learning on large-scale noisy data are becoming increasingly popular for zero-shot recognition problems. In this paper we improve the following three aspects of the contrastive pre-training pipeline: dataset noise, model initialization and the training objective. First, we propose a straightforward filtering strategy titled Complexity, Action, and Text-spotting (CAT) that significantly reduces dataset size, while achieving improved performance across zero-shot vision-language tasks. Next, we propose an approach titled Concept Distillation to leve…
Unsupervised person re-identification (Re-Id) has attracted increasing attention due to its practical application in the read-world video surveillance system. The traditional unsupervised Re-Id are mostly based on the method alternating between clustering and fine-tuning with the classification or metric learning objectives on the grouped clusters. However, since person Re-Id is an open-set problem, the clustering based methods often leave out lots of outlier instances or group the instances into the wrong clusters, thus they can not make full use of the training samples as a whole. To solve t…
Image-based object pose estimation sounds amazing because in real applications the shape of object is oftentimes not available or not easy to take like photos. Although it is an advantage to some extent, un-explored shape information in 3D vision learning problem looks like "flaws in jade". In this paper, we deal with the problem in a reasonable new setting, namely 3D shape is exploited in the training process, and the testing is still purely image-based. We enhance the performance of image-based methods for category-agnostic object pose estimation by exploiting 3D knowledge learned by a multi…
Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China（上海大学机械与自动化工程学院）；Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（香港城市大学计算机科学系）
Enriching Knowledge Distillation with Intra-Class Contrastive Learning
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其跨学科应用重点实验室（东南大学），教育部，中国）
针对知识蒸馏中教师软标签缺乏类内多样表示的问题，本文提出带 margin loss 的类内对比学习教师训练方法，并通过理论与实验验证其有效性。
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Google Research（谷歌研究院）
Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a stud…
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Data Science, University of Science and Technology of China（中国科学技术大学数据科学学院）；iFLYTEK AI Research（讯飞AI研究院）
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；University of Bern（伯尔尼大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
Recently, contrastive learning has achieved great results in self-supervised learning, where the main idea is to push two augmentations of an image (positive pairs) closer compared to other random images (negative pairs). We argue that not all random images are equal. Hence, we introduce a self supervised learning algorithm where we use a soft similarity for the negative images rather than a binary distinction between positive and negative pairs. We iteratively distill a slowly evolving teacher model to the student model by capturing the similarity of a query image to some random images and tr…
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（浙江大学-阿里巴巴前沿技术联合研究中心）；National University of Singapore（新加坡国立大学）
Model inversion, whose goal is to recover training data from a pre-trained model, has been recently proved feasible. However, existing inversion methods usually suffer from the mode collapse problem, where the synthesized instances are highly similar to each other and thus show limited effectiveness for downstream tasks, such as knowledge distillation. In this paper, we propose Contrastive Model Inversion~(CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue. Our main observation is that, under the constraint of the same amount …
Beyond Linear Bottlenecks: Spline-Based Knowledge Distillation for Culturally Diverse Art Style Classification
Institute of Applied Sciences；Intelligent Systems – CNR, Via per Monteroni, 73100 Lecce, Italy Department of Innovation Engineering, University of Salento UPV/EHU, University of the Basque Country, 20018 San Sebastian, Spain Université Polytechnique Hauts-de-France, Université de Lille, CNRS, 59313 Valenciennes, France
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）；Horizon Robotics（地平线机器人）
The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea of MCL is to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-n…
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
EKFZ for Digital Health（数字健康EKFZ）；TU Dresden（德累斯顿工业大学）；University of Augsburg（奥格斯堡大学）；Bavarian Cancer Research Center (BZKF)（巴伐利亚癌症研究中心）；Department of Medicine I, TU Dresden（德累斯顿工业大学第一医学院）；NCT Heidelberg（海德堡NCT）；University of Leeds（利兹大学）
Preview-based Category Contrastive Learning for Knowledge Distillation
School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Software, Shandong University（山东大学软件学院）；Alibaba Group（阿里巴巴集团）
A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation
Graduate School of Information Science and Technology, Hokkaido University（信息科学与技术研究生院，北海道大学）；Graduate School of Information Science and Technology, The University of Tokyo（信息科学与技术研究生院，东京大学）
University of Science and Technology of China（中国科学技术大学）；The University of Adelaide（阿德莱德大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Shanghai Artificial Intelligent Laboratory（上海人工智能实验室）
Recent progresses on self-supervised 3D human action representation learning are largely attributed to contrastive learning. However, in conventional contrastive frameworks, the rich complementarity between different skeleton modalities remains under-explored. Moreover, optimized with distinguishing self-augmented samples, models struggle with numerous similar positive instances in the case of limited action categories. In this work, we tackle the aforementioned problems by introducing a general Inter- and Intra-modal Mutual Distillation (I$^2$MD) framework. In I$^2$MD, we first re-formulate t…
Harbin Institute of Technology（哈尔滨工业大学）；JD Explore Academy at JD.com（京东集团京东探索研究院）；Eindhoven University of Technology（埃因霍温理工大学）
Although the vision-and-language pretraining (VLP) equipped cross-modal image-text retrieval (ITR) has achieved remarkable progress in the past two years, it suffers from a major drawback: the ever-increasing size of VLP models restricts its deployment to real-world search scenarios (where the high latency is unacceptable). To alleviate this problem, we present a novel plug-in dynamic contrastive distillation (DCD) framework to compress the large VLP models for the ITR task. Technically, we face the following two challenges: 1) the typical uni-modal metric learning approach is difficult to dir…
