Knowledge Distillation (KD) methods are capable of transferring the knowledge encoded in a large and complex teacher into a smaller and faster student. Early methods were usually limited to transferring the knowledge only between the last layers of the networks, while latter approaches were capable of performing multi-layer KD, further increasing the accuracy of the student. However, despite their improved performance, these methods still suffer from several limitations that restrict both their efficiency and flexibility. First, existing KD methods typically ignore that neural networks undergo through different learning phases during the training process, which often requires different types of supervision for each one. Furthermore, existing multi-layer KD methods are usually unable to effectively handle networks with significantly different architectures (heterogeneous KD). In this paper we propose a novel KD method that works by modeling the information flow through the various layers of the teacher model and then train a student model to mimic this information flow. The proposed method is capable of overcoming the aforementioned limitations by using an appropriate supervision scheme during the different phases of the training process, as well as by designing and training an appropriate auxiliary teacher model that acts as a proxy model capable of "explaining" the way the teacher works to the student. The effectiveness of the proposed method is demonstrated using four image datasets and several different evaluation setups.
Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University（复旦大学电磁波信息科学教育部重点实验室）；Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University（复旦大学微波视觉智能感知学科与技术中心）
HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning
Southern University of Science and Technology（南方科技大学）；Zhejiang University（浙江大学）；Technical University of Munich（慕尼黑工业大学）；City University of Hong Kong（香港城市大学）；Shanghai Jiao Tong University（上海交通大学）
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
School of information and Electronics, Beijing Institute of Technology（北京理工大学信息与电子学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；School of Computer Science, Faculty of Engineering, The University of Sydney（悉尼大学工程学院计算机科学学院）
Insight SFI Research Centre for Data Analytics（Insight SFI数据分析研究中心）；Dublin City University（都柏林城市大学）；University of Catania（卡塔尼亚大学）
Video saliency prediction has recently attracted attention of the research community, as it is an upstream task for several practical applications. However, current solutions are particularly computationally demanding, especially due to the wide usage of spatio-temporal 3D convolutions. We observe that, while different model architectures achieve similar performance on benchmarks, visual variations between predicted saliency maps are still significant. Inspired by this intuition, we propose a lightweight model that employs multiple simple heterogeneous decoders and adopts several practical app…
Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation
School of Computer and Mathematics, Central South University of Forestry and Technology（中南林业科技大学计算机与数学学院）；College of Computer Science and Technology, and the Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology（国防科技大学计算机科学与技术学院及前沿装备数字化软件实验室）
In this paper, we study the problem of unifying knowledge from a set of classifiers with different architectures and target classes into a single classifier, given only a generic set of unlabelled data. We call this problem Unifying Heterogeneous Classifiers (UHC). This problem is motivated by scenarios where data is collected from multiple sources, but the sources cannot share their data, e.g., due to privacy concerns, and only privately trained models can be shared. In addition, each source may not be able to gather data to train all classes due to data availability at each source, and may n…
Beijing University of Posts and Telecommunications（北京邮电大学）；ByteDance（字节跳动）；USTC（中国科学技术大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Key Laboratory of Interactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部互动技术与体验系统重点实验室）；Zhongguancun Academy（中关村科学城）
Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection
Faculty of Mathematics and Computer Science, University of Bucharest（布加勒斯特大学数学与计算机科学学院）；Gheorghe Mihoc-Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy（罗马尼亚科学院格奥尔基·米霍茨-凯奥斯·伊阿科夫学院数学统计与应用数学研究所）
University of Rochester（罗切斯特大学）；JD AI Research（京东人工智能研究院）
In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous anticipation tasks that aim at action label prediction, our work targets at generating natural language outputs that provide interpretable and accurate descriptions of future action steps. It is a challenging task due to the lack of semantic information extracted from the instructional videos. To overcome this challenge, we propose a novel knowledge distillation framework to exploit the related external textual knowledge …
East China Normal University（华东师范大学）；Huawei Technologies（华为技术有限公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xiamen University（厦门大学）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen University（深圳大学）；University of Sydney（悉尼大学）
Information Bottleneck (IB) based multi-view learning provides an information theoretic principle for seeking shared information contained in heterogeneous data descriptions. However, its great success is generally attributed to estimate the multivariate mutual information which is intractable when the network becomes complicated. Moreover, the representation learning tradeoff, {\it i.e.}, prediction-compression and sufficiency-consistency tradeoff, makes the IB hard to satisfy both requirements simultaneously. In this paper, we design several variational information bottlenecks to exploit two…
Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-level features like raw pixel RGB values. In this paper, we propose masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning: firstly we pretrain an image (or video) model by recovering low-level features of masked patches, then we use the resulting features as targets for masked feature modeling. For …
The performance of federated learning in neural networks is generally influenced by the heterogeneity of the data distribution. For a well-performing global model, taking a weighted average of the local models, as done by most existing federated learning algorithms, may not guarantee consistency with local models in the space of neural network maps. In this paper, we propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) (i.e., without data on the server). The FedDKD introduces a module of decentralized knowledge distillation …
KAIST（韩国科学技术院）；Institute for Basic Science（基础科学研究所）；Microsoft Research Asia（微软亚洲研究院）；Tsinghua University（清华大学）
This paper presents FedX, an unsupervised federated learning framework. Our model learns unbiased representation from decentralized and heterogeneous local data. It employs a two-sided knowledge distillation with contrastive learning as a core component, allowing the federated system to function without requiring clients to share any data features. Furthermore, its adaptable architecture can be used as an add-on module for existing unsupervised algorithms in federated settings. Experiments show that our model improves performance significantly (1.58--5.52pp) on five unsupervised algorithms.
Knowledge distillation (KD) is a new method for transferring knowledge of a structure under training to another one. The typical application of KD is in the form of learning a small model (named as a student) by soft labels produced by a complex model (named as a teacher). Due to the novel idea introduced in KD, recently, its notion is used in different methods such as compression and processes that are going to enhance the model accuracy. Although different techniques are proposed in the area of KD, there is a lack of a model to generalize KD techniques. In this paper, various studies in the …
Dealing With Heterogeneous 3D MR Knee Images: A Federated Few-Shot Learning Method With Dual Knowledge Distillation
Rutgers University（罗格斯大学）；Sichuan University West China Hospital（四川大学华西医院）；Walmart Global Tech（沃尔玛全球科技）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Tongren Hospital（同仁医院）；Fudan University（复旦大学）；Zhongshan Hospital（中山医院）；SenseTime Research（商汤科技研究院）
Shandong University（山东大学）；National University of Singapore（新加坡国立大学）；Tencent AI Lab（腾讯人工智能实验室）
Recently, the booming fashion sector and its huge potential benefits have attracted tremendous attention from many research communities. In particular, increasing research efforts have been dedicated to the complementary clothing matching as matching clothes to make a suitable outfit has become a daily headache for many people, especially those who do not have the sense of aesthetics. Thanks to the remarkable success of neural networks in various applications such as image classification and speech recognition, the researchers are enabled to adopt the data-driven learning methods to analyze fa…
Knowledge distillation (KD) has become a well established paradigm for compressing deep neural networks. The typical way of conducting knowledge distillation is to train the student network under the supervision of the teacher network to harness the knowledge at one or multiple spots (i.e., layers) in the teacher network. The distillation spots, once specified, will not change for all the training samples, throughout the whole distillation process. In this work, we argue that distillation spots should be adaptive to training samples and distillation epochs. We thus propose a new distillation s…
Decentralized learning with private data is a central problem in machine learning. We propose a novel distillation-based decentralized learning technique that allows multiple agents with private non-iid data to learn from each other, without having to share their data, weights or weight updates. Our approach is communication efficient, utilizes an unlabeled public dataset and uses multiple auxiliary heads for each client, greatly improving training efficiency in the case of heterogeneous data. This approach allows individual models to preserve and enhance performance on their private tasks whi…
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology（澳大利亚昆士兰科技大学电气工程与机器人学院SAIVT组）；CSIRO Robotics, CSIRO（澳大利亚联邦科学与工业研究组织机器人部）
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
School of Artificial Intelligence and Computer Science, Shaanxi Normal University（陕西师范大学人工智能与计算机科学学院）；iFLYTEK Research, iFLYTEK Co., Ltd.（科大讯飞股份有限公司）；Key Laboratory of Intelligent Computing and Service Technology for Folk SongMinistry of Culture and Tourism（文化和旅游部民歌智能计算与服务技术重点实验室）
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Department of Electrical Engineering Indian Institute of Technology Bombay 21D070002（电子工程系 印度理工学院班加罗尔 21D070002）；Department of Electrical Engineering Indian Institute of Technology Bombay 210010010（电子工程系 印度理工学院班加罗尔 210010010）；Department of Electrical Engineering Indian Institute of Technology Bombay 210020023（电子工程系 印度理工学院班加罗尔 210020023）
Beihang University（北京航空航天大学）；SenseTime Research（商汤科技研究院）；ETH Zurich（苏黎世联邦理工学院）；Shandong University（山东大学）；University of Science and Technology of China（中国科学技术大学）
Conventional knowledge distillation (KD) methods for object detection mainly concentrate on homogeneous teacher-student detectors. However, the design of a lightweight detector for deployment is often significantly different from a high-capacity detector. Thus, we investigate KD among heterogeneous teacher-student pairs for a wide application. We observe that the core difficulty for heterogeneous KD (hetero-KD) is the significant semantic gap between the backbone features of heterogeneous detectors due to the different optimization manners. Conventional homogeneous KD (homo-KD) methods suffer …
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Semantic segmentation for SAR (Synthetic Aperture Radar) images has attracted increasing attention in the remote sensing community recently, due to SAR's all-time and all-weather imaging capability. However, SAR images are generally more difficult to be segmented than their EO (Electro-Optical) counterparts, since speckle noises and layovers are inevitably involved in SAR images. To address this problem, we investigate how to introduce EO features to assist the training of a SAR-segmentation model, and propose a heterogeneous feature distillation network for segmenting SAR images, called HFD-N…
Socialized Detector Learning: Trajectory-Guided and Reciprocal Distillation for Heterogeneous Object Detectors
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；School of Sports Training, Tianjin University of Sport（天津体育学院运动训练学院）；Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Electronic Science, National University of Defense Technology（国防科技大学电子科学学院）；School of Automation, Southeast University（东南大学自动化学院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
Institute of Computer Science and Technology, Peking University（北京大学计算机科学技术研究所）
Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the tr…
