Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual, between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04$\%$, 23.14$\%$, 53.59$\%$, and 56.86$\%$ of the teachers' computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.
The University of Hong Kong（香港大学）；ByteDance Inc.（字节跳动公司）
In this paper, we empirically study how to make the most of low-resolution frames for efficient video recognition. Existing methods mainly focus on developing compact networks or alleviating temporal redundancy of video inputs to increase efficiency, whereas compressing frame resolution has rarely been considered a promising solution. A major concern is the poor recognition accuracy on low-resolution frames. We thus start by analyzing the underlying causes of performance degradation on low-resolution frames. Our key finding is that the major cause of degradation is not information loss in the …
It is challenging to bridge the performance gap between Binary CNN (BCNN) and Floating point CNN (FCNN). We observe that, this performance gap leads to substantial residuals between intermediate feature maps of BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This training strategy, i.e., optimizing each binary convolutional block with block-wise distillation loss derived from FCNN, leads to a more effective optimization to BCNN. It also motivates us to update the binary convolutional block architecture to facil…
Knowledge distillation (KD) is generally considered as a technique for performing model compression and learned-label smoothing. However, in this paper, we study and investigate the KD approach from a new perspective: we study its efficacy in training a deeper network without any residual connections. We find that in most of the cases, non-residual student networks perform equally or better than their residual versions trained on raw data without KD (baseline network). Surprisingly, in some cases, they surpass the accuracy of baseline networks even with the inferior teachers. After a certain d…
Knowledge distillation compacts deep networks by letting a small student network learn from a large teacher network. The accuracy of knowledge distillation recently benefited from adding residual layers. We propose to reduce the size of the student network even further by recasting multiple residual layers in the teacher network into a single recurrent student layer. We propose three variants of adding recurrent connections into the student network, and show experimentally on CIFAR-10, Scenes and MiniPlaces, that we can reduce the number of parameters at little loss in accuracy.
Expandable Residual Approximation for Knowledge Distillation
School of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术学院，哈尔滨工业大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；School of Electronics, Electrical and Communication Engineering, University of Chinese Academy of Sciences（电子、电气与通信工程学院，中国科学院大学）
本文提出可扩展残差逼近（ERA）方法，通过多分支残差网络分步分解残差知识并整合教师权重，有效缓解师生容量差距，在 ImageNet 和 MS COCO 等视觉任务上取得显著提升。
ATMS-KD: Adaptive Temperature and Mixed Sample Knowledge Distillation for a Lightweight Residual CNN in Agricultural Embedded Systems
TEDAEEP Research Group, FPL, Abdelmalek saadi University（TEDAEEP研究组，FPL，阿德莱尔马萨迪大学）；Smart System Laboratory (SSL), ENSIAS, Mohammed V University（智能系统实验室（SSL），ENSIAS，穆罕默德五世大学）
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（新加坡国立大学）；National University of Defense Technology（中国人民解放军国防科技大学）；Pensees Pte Ltd（Pensees私人有限公司）；University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Recent deep learning based face recognition methods have achieved great performance, but it still remains challenging to recognize very low-resolution query face like 28x28 pixels when CCTV camera is far from the captured subject. Such face with very low-resolution is totally out of detail information of the face identity compared to normal resolution in a gallery and hard to find corresponding faces therein. To this end, we propose a Resolution Invariant Model (RIM) for addressing such cross-resolution face recognition problems, with three distinct novelties. First, RIM is a novel and unified…
Tianjin University（天津大学）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Group Limited（商汤科技集团有限公司）；Nanyang Technological University（南洋理工大学）
Knowledge distillation (KD) is one of the most potent ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this work proposes Residual Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant (A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In th…
Filter level pruning is an effective method to accelerate the inference speed of deep CNN models. Although numerous pruning algorithms have been proposed, there are still two open issues. The first problem is how to prune residual connections. We propose to prune both channels inside and outside the residual connections via a KL-divergence based criterion. The second issue is pruning with limited data. We observe an interesting phenomenon: directly pruning on a small dataset is usually worse than fine-tuning a small model which is pruned or trained from scratch on the large dataset. Knowledge …
Attention Residual Fusion Network with Contrast for Source-free Domain Adaptation
Faculty of Military Health Services, Naval Medical University (Second Military Medical University)（海军医学大学（第二军医大学）军事卫生服务学院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；KLATASDS-MOE
CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillation
College of Electronic and Information Engineering, Tongji University, Shanghai, China（电子信息工程学院，同济大学，上海，中国）；State Key Laboratory of Autonomous Intelligent Unmanned Systems（自主智能无人系统国家重点实验室）
Canon Information Technology (Beijing) Co., LTD（佳能信息技术（北京）有限公司）；Canon Inc.（佳能公司）
Despite the recent works on knowledge distillation (KD) have achieved a further improvement through elaborately modeling the decision boundary as the posterior knowledge, their performance is still dependent on the hypothesis that the target network has a powerful capacity (representation ability). In this paper, we propose a knowledge representing (KR) framework mainly focusing on modeling the parameters distribution as prior knowledge. Firstly, we suggest a knowledge aggregation scheme in order to answer how to represent the prior knowledge from teacher network. Through aggregating the param…
MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection
National Yang Ming Chiao Tung University（国立阳明交通大学）；University of Washington（华盛顿大学）；University of Southern California（南加州大学）；Cisco Systems, Inc.（思科系统公司）；Taijitu AI, Inc.（太极图人工智能公司）；National Taiwan Normal University（国立台湾师范大学）；National Taiwan University（国立台湾大学）
TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
CUNY City College of NY（纽约城市大学城市学院）；Wyze Labs, Inc.（威智实验室公司）；University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）
Hefei University of Technology（合肥工业大学）；Jianghuai Advance Technology Center（江淮先进技术中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）；Kaiyang Laboratory, Chery（凯阳实验室，中国奇瑞）；Zhejiang University of Technology（浙江工业大学）
We propose a method to infer a dense depth map from a single image, its calibration, and the associated sparse point cloud. In order to leverage existing models (teachers) that produce putative depth maps, we propose an adaptive knowledge distillation approach that yields a positive congruent training process, wherein a student model avoids learning the error modes of the teachers. In the absence of ground truth for model selection and training, our method, termed Monitored Distillation, allows a student to exploit a blind ensemble of teachers by selectively learning from predictions that best…
Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are generated from a generator, while no data are used in training the generator and in quantization. To this end, we propose data-free adversarial knowledge distillation, which minimizes the maximum distan…
VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
Shanghai Jiao Tong University（上海交通大学）；Xiaohongshu Inc.（小红书公司）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Southeast University（东南大学）
Indian Institute of Technology, Roorkee（印度理工学院鲁尔基分校）
This effort aims to reproduce the results of experiments and analyze the robustness of the review framework for knowledge distillation introduced in the CVPR '21 paper 'Distilling Knowledge via Knowledge Review' by Chen et al. Previous works in knowledge distillation only studied connections paths between the same levels of the student and the teacher, and cross-level connection paths had not been considered. Chen et al. propose a new residual learning framework to train a single student layer using multiple teacher layers. They also design a novel fusion module to condense feature maps across…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；TAL Education Group（好未来教育集团）
Modern object detection methods based on convolutional neural network suffer from severe catastrophic forgetting in learning new classes without original data. Due to time consumption, storage burden and privacy of old data, it is inadvisable to train the model from scratch with both old and new data when new object classes emerge after the model trained. In this paper, we propose a novel incremental object detector based on Faster R-CNN to continuously learn from new object classes without using old data. It is a triple network where an old model and a residual model as assistants for helping…
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
UNSW Sydney（新南威尔士大学悉尼分校）；University of Technology Sydney（技术大学悉尼分校）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Stanford University（斯坦福大学）；Shanghai Jiao Tong University（上海交通大学）
One-Step Residual Shifting Diffusion for Image Super-Resolution via Distillation
Kandinsky Lab（坎迪斯基实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Luzin Research Center（卢津研究所）；Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究 institute）；Applied AI Institute（应用人工智能研究所）
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
We push forward neural network compression research by exploiting a novel challenging task of large-scale conditional generative adversarial networks (GANs) compression. To this end, we propose a gradually shrinking GAN (PPCD-GAN) by introducing progressive pruning residual block (PP-Res) and class-aware distillation. The PP-Res is an extension of the conventional residual block where each convolutional layer is followed by a learnable mask layer to progressively prune network parameters as training proceeds. The class-aware distillation, on the other hand, enhances the stability of training b…
Recent advances in single image super-resolution (SISR) explored the power of convolutional neural network (CNN) to achieve a better performance. Despite the great success of CNN-based methods, it is not easy to apply these methods to edge devices due to the requirement of heavy computation. To solve this problem, various fast and lightweight CNN models have been proposed. The information distillation network is one of the state-of-the-art methods, which adopts the channel splitting operation to extract distilled features. However, it is not clear enough how this operation helps in the design …
MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ETH Zurich（苏黎世联邦理工学院）；Institute of Artificial Intelligence, Beihang University（北航人工智能研究院）；Zhongguancun Laboratory（中关村实验室）；Department of Electrical Engineering, The City College（城市学院电子工程系）；Department of Computer Science, the Graduate Center, the City University of New York（纽约城市大学计算机科学系）；School of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与控制工程学院）
Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed coun…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；University of Chinese Academy of Sciences（中国科学院大学）
The balance between high accuracy and high speed has always been a challenging task in semantic image segmentation. Compact segmentation networks are more widely used in the case of limited resources, while their performances are constrained. In this paper, motivated by the residual learning and global aggregation, we propose a simple yet general and effective knowledge distillation framework called double similarity distillation (DSD) to improve the classification accuracy of all existing compact networks by capturing the similarity knowledge in pixel and category dimensions, respectively. Sp…
Nanjing University of Posts and Telecommunications（南京邮电大学）；The Chinese University of Hong Kong（香港中文大学）；Kyushu Institute of Technology（九州工业大学）；National Institute of Informatics（国立信息学研究所）
Convolutional neural networks based single-image super-resolution (SISR) has made great progress in recent years. However, it is difficult to apply these methods to real-world scenarios due to the computational and memory cost. Meanwhile, how to take full advantage of the intermediate features under the constraints of limited parameters and calculations is also a huge challenge. To alleviate these issues, we propose a lightweight yet efficient Feature Distillation Interaction Weighted Network (FDIWN). Specifically, FDIWN utilizes a series of specially designed Feature Shuffle Weighted Groups (…
