Robustness and Diversity Seeking Data-Free Knowledge Distillation
Northeastern University（东北大学）；Deakin University（迪肯大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Pengchao Han, Jihong Park, Shiqiang Wang, Yejun Liu
Knowledge distillation (KD) has enabled remarkable progress in model compression and knowledge transfer. However, KD requires a large volume of original data or their representation statistics that are not usually available in practice. Data-free KD has recently been proposed to resolve this problem, wherein teacher and student models are fed by a synthetic sample generator trained from the teacher. Nonetheless, existing data-free KD methods rely on fine-tuning of weights to balance multiple losses, and ignore the diversity of generated samples, resulting in limited accuracy and robustness. To overcome this challenge, we propose robustness and diversity seeking data-free KD (RDSKD) in this paper. The generator loss function is crafted to produce samples with high authenticity, class diversity, and inter-sample diversity. Without real data, the objectives of seeking high sample authenticity and class diversity often conflict with each other, causing frequent loss fluctuations. We mitigate this by exponentially penalizing loss increments. With MNIST, CIFAR-10, and SVHN datasets, our experiments show that RDSKD achieves higher accuracy with more robustness over different hyperparameter settings, compared to other data-free KD methods such as DAFL, MSKD, ZSKD, and DeepInversion.
Virtual Reality and Intelligent Systems Research Institute, Hangzhou Normal University（杭州师范大学虚拟现实与智能系统研究院）；School of Information Science & Engineering, Hangzhou Normal University（杭州师范大学信息科学与工程学院）；Alibaba business school, Hangzhou Normal University（杭州师范大学阿里巴巴商学院）；College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）
Knowledge distillation is an effective method to transfer the knowledge from the cumbersome teacher model to the lightweight student model. Online knowledge distillation uses the ensembled prediction results of multiple student models as soft targets to train each student model. However, the homogenization problem will lead to difficulty in further improving model performance. In this work, we propose a new distillation method to enhance the diversity among multiple student models. We introduce Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the networ…
Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustment
Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Institute of High Performance Computing（高性能计算研究所）；XiDian University（西安电子科技大学）；National University of Singapore（新加坡国立大学）；Hubei University（湖北大学）
Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Shandong Computer Science Center（山东省计算机科学中心）；Spatialtemporal AI（时空人工智能公司）；Tongji University（同济大学）
Soft Label Pruning and Quantization for Large-Scale Dataset Distillation
CFAR, Agency for Science, Technology and Research, Singapore（新加坡科技研究局应用科学与技术研究院）；IHPC, Agency for Science, Technology and Research, Singapore（新加坡科技研究局信息与通信技术研究院）；National University of Singapore（新加坡国立大学）
Imperial College London（帝国理工学院）；University of Amsterdam（阿姆斯特丹大学）；ETH Zurich（苏黎世联邦理工学院）；University of Warsaw（华沙大学）；Google Research（谷歌研究院）；University of California, Irvine（加州大学欧文分校）
Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden of the ensemble while trying to preserve its predictive behavior. Most existing distillation formulations summarize the ensemble by capturing its average predictions. As a result, the diversity of the ensemble predictions, stemming from each member, is lost. Thus, the distilled mo…
Recently, researchers have shown an increased interest in the online knowledge distillation. Adopting an one-stage and end-to-end training fashion, online knowledge distillation uses aggregated intermediated predictions of multiple peer models for training. However, the absence of a powerful teacher model may result in the homogeneity problem between group peers, affecting the effectiveness of group distillation adversely. In this paper, we propose a novel online knowledge distillation method, \textbf{C}hannel \textbf{S}elf-\textbf{S}upervision for Online Knowledge Distillation (CSS), which st…
Knowledge Distillation has shown very promising abil-ity in transferring learned representation from the largermodel (teacher) to the smaller one (student).Despitemany efforts, prior methods ignore the important role ofretaining inter-channel correlation of features, leading tothe lack of capturing intrinsic distribution of the featurespace and sufficient diversity properties of features in theteacher network.To solve the issue, we propose thenovel Inter-Channel Correlation for Knowledge Distillation(ICKD), with which the diversity and homology of the fea-ture space of the student network can …
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（浙江大学-阿里巴巴前沿技术联合研究中心）；National University of Singapore（新加坡国立大学）
Model inversion, whose goal is to recover training data from a pre-trained model, has been recently proved feasible. However, existing inversion methods usually suffer from the mode collapse problem, where the synthesized instances are highly similar to each other and thus show limited effectiveness for downstream tasks, such as knowledge distillation. In this paper, we propose Contrastive Model Inversion~(CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue. Our main observation is that, under the constraint of the same amount …
Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能技术国家重点实验室）；Guangdong Institute of Intelligence Science and Technology（广东省智能科学与技术研究院）；Institute of Collaborative Innovation, University of Macau（澳门大学协同创新研究院）；Beijing Huahang Institute of Radio Measurement（北京华航无线电测量研究所）
Synthetic Data Generation for Emotional Depth Faces: Optimizing Conditional DCGANs via Genetic Algorithms in the Latent Space and Stabilizing Training with Knowledge Distillation
Cyrus Intelligence Research Ltd（居鲁士智能研究有限公司）；Pars AI
Yonsei University（延世大学）；Seoul National University（首尔大学）
Active learning can be defined as iterations of data labeling, model training, and data acquisition, until sufficient labels are acquired. A traditional view of data acquisition is that, through iterations, knowledge from human labels and models is implicitly distilled to monotonically increase the accuracy and label consistency. Under this assumption, the most recently trained model is a good surrogate for the current labeled data, from which data acquisition is requested based on uncertainty/diversity. Our contribution is debunking this myth and proposing a new objective for distillation. Fi…
Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；St Vincent’s Institute of Medical Research（圣文森特医学研究所）；St Vincent’s Hospital Melbourne（墨尔本圣文森特医院）
State-of-the-art (SOTA) deep learning mammogram classifiers, trained with weakly-labelled images, often rely on global models that produce predictions with limited interpretability, which is a key barrier to their successful translation into clinical practice. On the other hand, prototype-based models improve interpretability by associating predictions with training image prototypes, but they are less accurate than global models and their prototypes tend to have poor diversity. We address these two issues with the proposal of BRAIxProtoPNet++, which adds interpretability to a global model by e…
Progressive trajectory matching for medical dataset distillation
National Institute of Health Data Science, Peking University（北京大学健康数据科学研究院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究院）
Rethinking Reverse KL as Adaptive Entropy Distillation
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Science and Technology, Hong Kong Metropolitan University（香港都会大学科技学院）
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
University of Georgia（佐治亚大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Harvard University（哈佛大学）；Carnegie Mellon University（卡内基梅隆大学）；Vanderbilt University（范德堡大学）；Mayo Clinic Arizona（亚利桑那梅奥诊所）；Augusta University（奥古斯塔大学）
Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?
CFAR, Agency for Science, Technology and Research, Singapore（新加坡科技研究局CFAR）；National University of Singapore（新加坡国立大学）；IHPC, Agency for Science, Technology and Research, Singapore（新加坡科技研究局IHPC）
Distilling knowledge from huge pre-trained networks to improve the performance of tiny networks has favored deep learning models to be used in many real-time and mobile applications. Several approaches that demonstrate success in this field have made use of the true training dataset to extract relevant knowledge. In absence of the True dataset, however, extracting knowledge from deep networks is still a challenge. Recent works on data-free knowledge distillation demonstrate such techniques on classification tasks. To this end, we explore the task of data-free knowledge distillation for segment…
Zhejiang University（浙江大学）；Hangzhou City University（杭州城市大学）；Zhejiang Lab（之江实验室）
Most existing online knowledge distillation(OKD) techniques typically require sophisticated modules to produce diverse knowledge for improving students' generalization ability. In this paper, we strive to fully utilize multi-model settings instead of well-designed modules to achieve a distillation effect with excellent generalization performance. Generally, model generalization can be reflected in the flatness of the loss landscape. Since averaging parameters of multiple models can find flatter minima, we are inspired to extend the process to the sampled convex combinations of multi-student mo…
Knowledge distillation (KD) achieves promising results on the challenging problem of unsupervised anomaly detection (AD).The representation discrepancy of anomalies in the teacher-student (T-S) model provides essential evidence for AD. However, using similar or identical architectures to build the teacher and student models in previous studies hinders the diversity of anomalous representations. To tackle this problem, we propose a novel T-S model consisting of a teacher encoder and a student decoder and introduce a simple yet effective "reverse distillation" paradigm accordingly. Instead of re…
East China Normal University（华东师范大学）；Huawei Technologies（华为技术有限公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiamen University（厦门大学）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen University（深圳大学）；University of Sydney（悉尼大学）
Information Bottleneck (IB) based multi-view learning provides an information theoretic principle for seeking shared information contained in heterogeneous data descriptions. However, its great success is generally attributed to estimate the multivariate mutual information which is intractable when the network becomes complicated. Moreover, the representation learning tradeoff, {\it i.e.}, prediction-compression and sufficiency-consistency tradeoff, makes the IB hard to satisfy both requirements simultaneously. In this paper, we design several variational information bottlenecks to exploit two…
Zhejiang University（浙江大学）；ZJU-LianlianPay Joint Research Center（浙江大学-连连支付联合研究中心）；Zhejiang University of Technology（浙江工业大学）
Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate predictions of multiple student models as targets to train each student model. Although group-derived targets give a good recipe for teacher-free distillation, group members are homogenized quickly with simple aggregation functions, leading to early saturated solutions. In this wor…
