Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.
Rethinking Data Distillation: Do Not Overlook Calibration
Texas A&M University（德州农工大学）；Zhejiang University（浙江大学）；Certik；University of Maryland, College Park（马里兰大学帕克分校）；Purdue University（普渡大学）；North Carolina State University（北卡罗来纳州立大学）
In this paper, we study \xw{dataset distillation (DD)}, from a novel perspective and introduce a \emph{dataset factorization} approach, termed \emph{HaBa}, which is a plug-and-play strategy portable to any existing DD baseline. Unlike conventional DD approaches that aim to produce distilled and representative samples, \emph{HaBa} explores decomposing a dataset into two components: data \emph{Ha}llucination networks and \emph{Ba}ses, where the latter is fed into the former to reconstruct image samples. The flexible combinations between bases and hallucination networks, therefore, equip the dist…
University of Liverpool（利物浦大学）；Huawei Ireland Research Center（华为爱尔兰研究中心）
Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel \emph{self-distillation} based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed {\em cross-correlation objective for self-distilled pruning} implicitly …
Carnegie Mellon University（卡内基梅隆大学）；Massachusetts Institute of Technology（麻省理工学院）；UC Berkeley（加州大学伯克利分校）
Dataset distillation is the task of synthesizing a small dataset such that a model trained on the synthetic set will match the test accuracy of the model trained on the full dataset. In this paper, we propose a new formulation that optimizes our distilled data to guide networks to a similar state as those trained on real data across many training steps. Given a network, we train it for several iterations on our distilled data and optimize the distilled data with respect to the distance between the synthetically trained parameters and the parameters trained on real data. To efficiently obtain t…
PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty
Zhejiang University（浙江大学）；Hokkaido University（北海道大学）；The University of Queensland（昆士兰大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Dalian University of Technology（大连理工大学）
University of Toronto（多伦多大学）；Toronto AI Lab（多伦多人工智能实验室）；LG Electronics Canada（LG电子加拿大公司）
Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achiev…
FocusDD: Real-World Scene Infusion for Robust Dataset Distillation
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Swiss Data Science Center（瑞士数据科学中心）；Graz University of Technology（格拉茨工业大学）；Complexity Science Hub Vienna（维也纳复杂科学中心）
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Amazon Alexa（亚马逊Alexa）；Harvard University（哈佛大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Language models excel at generating coherent text, and model compression techniques such as knowledge distillation have enabled their use in resource-constrained settings. However, these models can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions. Therefore, knowledge distillation without any fairness constraints may preserve or exaggerate the teacher model's biases onto the distilled model. To this end, we present a novel approach to mitigate gender disparity in text generation by learning a fair model during knowledge …
Google Research（谷歌研究院）；Verily Life Sciences（Verily生命科学公司）；Pacific Biosciences（太平洋生物科学公司）
Accurate genome sequencing can improve our understanding of biology and the genetic basis of disease. The standard approach for generating DNA sequences from PacBio instruments relies on HMM-based models. Here, we introduce Distilled DeepConsensus - a distilled transformer-encoder model for sequence correction, which improves upon the HMM-based methods with runtime constraints in mind. Distilled DeepConsensus is 1.3x faster and 1.5x smaller than its larger counterpart while improving the yield of high quality reads (Q30) over the HMM-based method by 1.69x (vs. 1.73x for larger model). With imp…
DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation
Huazhong University of Science and Technology（华中科技大学）；Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）；IHPC, A*STAR（新加坡科研局信息通信研究院）；University of Nevada, Reno（内华达大学雷诺分校）；Brookhaven National Laboratory（布鲁克海文国家实验室）
Simon Fraser University（西蒙弗雷泽大学）；Borealis AI（Borealis人工智能研究院）
Knowledge distillation is a widely applicable technique for training a student neural network under the guidance of a trained teacher network. For example, in neural network compression, a high-capacity teacher is distilled to train a compact student; in privileged learning, a teacher trained with privileged data is distilled to train a student without access to that data. The distillation loss determines how a teacher's knowledge is captured and transferred to the student. In this paper, we propose a new form of knowledge distillation loss that is inspired by the observation that semantically…
The University of Tokyo Hospital（东京大学医院）；Graduate School of Medicine, The University of Tokyo（东京大学医学院）；Junpukai Health Maintenance Center（顺会健康体检中心）；Hokkaido University（北海道大学）
Robust Dataset Distillation by Matching Adversarial Trajectories
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Microsoft Corporation（微软公司）；University of Wollongong（卧龙岗大学）
本文提出鲁棒数据集蒸馏任务及 MAT 方法，通过在轨迹生成中引入对抗样本，使自然训练的模型兼具准确率与对抗鲁棒性。
We propose a method to infer a dense depth map from a single image, its calibration, and the associated sparse point cloud. In order to leverage existing models (teachers) that produce putative depth maps, we propose an adaptive knowledge distillation approach that yields a positive congruent training process, wherein a student model avoids learning the error modes of the teachers. In the absence of ground truth for model selection and training, our method, termed Monitored Distillation, allows a student to exploit a blind ensemble of teachers by selectively learning from predictions that best…
Dataset Distillation via Adversarial Prediction Matching
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61所）；University of Electronic Science and Technology of China（电子科技大学）；Dongguan University of Technology（东莞理工学院）
Despite the success that metric learning based approaches have achieved in few-shot learning, recent works reveal the ineffectiveness of their episodic training mode. In this paper, we point out two potential reasons for this problem: 1) the random episodic labels can only provide limited supervision information, while the relatedness information between the query and support samples is not fully exploited; 2) the meta-learner is usually constrained by the limited contextual information of the local episode. To overcome these problems, we propose a new Global Relatedness Decoupled-Distillation…
Bayesian models have many desirable properties, most notable is their ability to generalize from limited data and to properly estimate the uncertainty in their predictions. However, these benefits come at a steep computational cost as Bayesian inference, in most cases, is computationally intractable. One popular approach to alleviate this problem is using a Monte-Carlo estimation with an ensemble of models sampled from the posterior. However, this approach still comes at a significant computational cost, as one needs to store and run multiple models at test time. In this work, we investigate h…
University of British Columbia（不列颠哥伦比亚大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored. However, they consume a lot of memory which poses a challenge for edge deployment. Embedding matrices, typically, contain most of the parameters for language models and about a third for machine translation systems. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low-rank matrix decomposition and knowledge distillation. First, we initialize the weights of our decomposed matrices by learning to reconstruct the full pre-traine…
Today, transformer language models serve as a core component for majority of natural language processing tasks. Industrial application of such models requires minimization of computation time and memory footprint. Knowledge distillation is one of approaches to address this goal. Existing methods in this field are mainly focused on reducing the number of layers or dimension of embeddings/hidden representations. Alternative option is to reduce the number of tokens in vocabulary and therefore the embeddings matrix of the student model. The main problem with vocabulary minimization is mismatch bet…
Shenzhen University of Advanced Technology（深圳先进技术大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Xi’an Jiaotong University（西安交通大学）
Feature Fusion and Knowledge-Distilled Multi-Modal Multi-Target Detection
School of Information and Communications, Hanoi University of Science and Technology（信息与通信学院，河内科学技术大学）；Telecom Neural Detection Lab, Polytechnique Montréal（电信神经检测实验室，蒙特利尔理工学院）
Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations. Conventional KD methods have concentrated on distilling knowledge in data units. To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed.…
Meta FAIR Labs（Meta FAIR 实验室）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (…
Grounding and Enhancing Informativeness and Utility in Dataset Distillation
EPIC Lab, SJTU（SJTU实验室）；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；Duke University（杜克大学）；The University of Chicago（芝加哥大学）
