Knowledge distillation(KD) aims to improve the performance of a student network by mimicing the knowledge from a powerful teacher network. Existing methods focus on studying what knowledge should be transferred and treat all samples equally during training. This paper introduces the adaptive sample weighting to KD. We discover that previous effective hard mining methods are not appropriate for distillation. Furthermore, we propose Prime-Aware Adaptive Distillation (PAD) by the incorporation of uncertainty learning. PAD perceives the prime samples in distillation and then emphasizes their effect adaptively. PAD is fundamentally different from and would refine existing methods with the innovative view of unequal training. For this reason, PAD is versatile and has been applied in various tasks including classification, metric learning, and object detection. With ten teacher-student combinations on six datasets, PAD promotes the performance of existing distillation methods and outperforms recent state-of-the-art methods.
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Knowledge distillation, which involves extracting the "dark knowledge" from a teacher network to guide the learning of a student network, has emerged as an essential technique for model compression and transfer learning. Unlike previous works that focus on the accuracy of student network, here we study a little-explored but important question, i.e., knowledge distillation efficiency. Our goal is to achieve a performance comparable to conventional knowledge distillation with a lower computation cost during training. We show that the UNcertainty-aware mIXup (UNIX) can serve as a clean yet effect…
HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning
Southern University of Science and Technology（南方科技大学）；Zhejiang University（浙江大学）；Technical University of Munich（慕尼黑工业大学）；City University of Hong Kong（香港城市大学）；Shanghai Jiao Tong University（上海交通大学）
Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Waterloo（滑铁卢大学）
Knowledge distillation (KD) has been proven to be useful for training compact object detection models. However, we observe that KD is often effective when the teacher model and student counterpart share similar proposal information. This explains why existing KD methods are less effective for 1-bit detectors, caused by a significant information discrepancy between the real-valued teacher and the 1-bit student. This paper presents an Information Discrepancy-aware strategy (IDa-Det) to distill 1-bit detectors that can effectively eliminate information discrepancies and significantly reduce the p…
Uncertainty-Aware Retinal Vessel Segmentation via Ensemble Distillation
University of Ibadan（伊巴迪安大学）；African Institute for Mathematical Sciences（非洲数学科学研究所）；University College London（伦敦大学学院）；Hertie Institute for Brain Health（海德研究所）
Australian National University（澳大利亚国立大学）；Data61-CSIRO Australia（澳大利亚联邦科学与工业研究组织Data61）；North South University Bangladesh（孟加拉国南北大学）；Monash University Australia（澳大利亚莫纳什大学）
Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation pro…
ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentation
McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston (UTHealth Houston)（麦威廉生物医学信息学学院，德克萨斯大学健康科学中心休斯顿分校（UTHealth休斯顿））
Key Laboratory for Information Science of Electromagnetic Waves, Ministry of Education, Fudan University（复旦大学电磁波信息科学教育部重点实验室）；Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University（复旦大学微波视觉智能感知学科与技术中心）
Uncertainty-Aware Knowledge Distillation for Compact and Efficient 6DoF Pose Estimation
SnT, University of Luxembourg（卢森堡大学SnT分校）；Cristal Lab, ENSI, University of Manouba（曼努巴大学ENSI的Cristal实验室）；Infinite Orbits, Toulouse（图卢兹的Infinite Orbits）
The linear ensemble based strategy, i.e., averaging ensemble, has been proposed to improve the performance in unsupervised domain adaptation tasks. However, a typical UDA task is usually challenged by dynamically changing factors, such as variable weather, views, and background in the unlabeled target domain. Most previous ensemble strategies ignore UDA's dynamic and uncontrollable challenge, facing limited feature representations and performance bottlenecks. To enhance the model, adaptability between domains and reduce the computational cost when deploying the ensemble model, we propose a nov…
Knowledge distillation aims to transfer knowledge to the student model by utilizing the predictions/features of the teacher model, and feature-based distillation has recently shown its superiority over logit-based distillation. However, due to the cumbersome computation and storage of extra feature transformation, the training overhead of feature-based methods is much higher than that of logit-based distillation. In this work, we revisit the logit-based knowledge distillation, and observe that the existing logit-based distillation methods treat the prediction logits only in the instance level,…
Pool-Select-Refine for Allocation-Aware Generative Dataset Distillation
Graduate School of Engineering, University of Fukui（福井大学工学研究科）；College of Computer Science and Artificial Intelligence, Southwest Minzu University（西南民族大学计算机科学与人工智能学院）
Centerness-based Instance-aware Knowledge Distillation with Task-wise Mutual Lifting for Object Detection on Drone Imagery
Beihang University（北京航空航天大学）；State Key Laboratory of Software Development Environment（软件开发环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
National University of Defense Technology（国防科技大学）；University of Oulu（奥卢大学）
Given a model well-trained with a large-scale base dataset, Few-Shot Class-Incremental Learning (FSCIL) aims at incrementally learning novel classes from a few labeled samples by avoiding overfitting, without catastrophically forgetting all encountered classes previously. Currently, semi-supervised learning technique that harnesses freely-available unlabeled data to compensate for limited labeled data can boost the performance in numerous vision tasks, which heuristically can be applied to tackle issues in FSCIL, i.e., the Semi-supervised FSCIL (Semi-FSCIL). So far, very limited work focuses o…
Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation
School of Computer and Mathematics, Central South University of Forestry and Technology（中南林业科技大学计算机与数学学院）；College of Computer Science and Technology, and the Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology（国防科技大学计算机科学与技术学院及前沿装备数字化软件实验室）
Visual Place Recognition in Forests with Depth-Aware Distillation
CSIRO Robotics, Brisbane, Australia（澳大利亚联邦科学与工业研究组织机器人实验室，布里斯班，澳大利亚）；University of Queensland, Brisbane, Australia（昆士兰大学，布里斯班，澳大利亚）；Queensland University of Technology, Brisbane, Australia（昆士兰科技大学，布里斯班，澳大利亚）
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam（河内科技大学电气与电子工程学院，越南）；Signal Analysis and Interpretation Lab, University of Southern California, United States（南加州大学信号分析与解释实验室，美国）
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Department of Electrical Engineering Indian Institute of Technology Bombay 21D070002（电子工程系 印度理工学院班加罗尔 21D070002）；Department of Electrical Engineering Indian Institute of Technology Bombay 210010010（电子工程系 印度理工学院班加罗尔 210010010）；Department of Electrical Engineering Indian Institute of Technology Bombay 210020023（电子工程系 印度理工学院班加罗尔 210020023）
Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences (UCAS)（中国科学院大学先进交叉科学学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Ant Group（蚂蚁集团）；Sangfor Technologies Inc.（深信服科技股份有限公司）
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
Seoul National University（首尔国立大学）；Qualcomm AI Research（高通人工智能研究院）；Qualcomm Technologies, Inc.（高通技术公司）
Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-…
Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics
CFAR, Agency for Science, Technology and Research (A*STAR), Singapore（科技研究局（A*STAR）新加坡分部）；IHPC, Agency for Science, Technology and Research (A*STAR), Singapore（科技研究局（A*STAR）新加坡信息与通信技术中心）；School of EEE, Nanyang Technological University, Singapore（南洋理工大学电子工程学院）；CCDS, Nanyang Technological University, Singapore（南洋理工大学计算机科学与工程系）
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
Beijing University of Posts and Telecommunications（北京邮电大学）；ByteDance（字节跳动）；USTC（中国科学技术大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Key Laboratory of Interactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部互动技术与体验系统重点实验室）；Zhongguancun Academy（中关村科学城）
Foreground-Aware Dataset Distillation via Dynamic Patch Selection
Graduate School of Information Science and Technology, Hokkaido University（信息科学与技术研究生学校，北海道大学）；Education and Research Center for Mathematical and Data Science, Hokkaido University（数学与数据科学教育与研究中心，北海道大学）；Faculty of Information Science and Technology, Hokkaido University（信息科学与技术学部，北海道大学）
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）
Tecnologico de Monterrey（蒙特雷理工学院）；University of Leeds（利兹大学）
Prevalence of gastrointestinal (GI) cancer is growing alarmingly every year leading to a substantial increase in the mortality rate. Endoscopic detection is providing crucial diagnostic support, however, subtle lesions in upper and lower GI are quite hard to detect and cause considerable missed detection. In this work, we leverage deep learning to develop a framework to improve the localization of difficult to detect lesions and minimize the missed detection rate. We propose an end to end student-teacher learning setup where class probabilities of a trained teacher model on one class with larg…
TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
CUNY City College of NY（纽约城市大学城市学院）；Wyze Labs, Inc.（威智实验室公司）；University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）
Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading
School of Public Health, Zhejiang University（浙江大学公共卫生学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Liangzhu Laboratory and WeDoctor Cloud（良渚实验室和WeDoctor云）；State Key Laboratory of Transvascular Implantation Devices, The Second Affiliated Hospital Zhejiang University School of Medicine（浙江大学医学院第二附属医院心血管植入器械国家重点实验室）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
Ground Reaction Force Estimation via Time-aware Knowledge Distillation
Department of Computer Science and Engineering, Seoul National University of Science and Technology（首尔科学技术大学计算机科学与工程系）；Geometric Media Lab, School of Arts, Media and Engineering and School of Electrical, Computer and Energy Engineering, Arizona State University（亚利桑那州立大学几何媒体实验室、艺术、媒体与工程学院和电气、计算机与能源工程学院）；School for Engineering of Matter, Transport and Energy, Arizona State University（亚利桑那州立大学工程科学学院）；Department of Computer Science and Engineering, University of Nevada, Reno（内华达大学里诺分校计算机科学与工程系）
MAD: Microenvironment-Aware Distillation -- A Pretraining Strategy for Virtual Spatial Omics from Microscopy
Research Laboratory of Electronics, Massachusetts Institute of Technology（麻省理工学院电子研究实验室）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；The Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology（佐治亚理工学院Wallace H. Coulter生物医学工程系）；H. Milton Stewart School of Industrial & Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H. Milton Stewart工业与系统工程学院）
