Displacement Preserving Relational Distillation for Robust Medical Segmentation
Bowling Green State University（鲍灵格林州立大学）；The University of Findlay（芬德利大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；University of Toronto（多伦多大学）；University of Houston - Clear Lake（休斯顿大学清水湖分校）；UTHealth Houston（德克萨斯大学健康科学中心休斯顿分校）
Zhicheng Ding, Xinyu Chu, Jung Im Choi, Qing Tian, Tianyu Shi, Xiaoqian Jiang, Lijing Zhu, Qizhen Lan
Accurate 3D medical segmentation is limited by anatomical variability and high computational costs. While knowledge distillation (KD) offers a route for model compression, conventional methods often fail to preserve complex structures and are overwhelmed by background noise. We propose Displacement-Preserving Relational Distillation (DPRD), which distills latent anatomical trajectories via vector based alignment to preserve the orientation and relative scale of the teacher's manifold, and prevents signal dilution by anchoring distillation in task-relevant structures. Integrated into nnU-Net, DPRD outperforms established baselines on ISLES 2022 and AMOS 2022 benchmarks. Notably, on the AMOS dataset, DPRD achieves a Dice score of 85.46%, edging out the high-capacity MedNeXt teacher while significantly reducing boundary errors. Despite utilizing only ~5% of the teacher's parameters and ~3% of its FLOPs, our approach maintains high structural consistency. This provides a robust, efficient solution for deploying high performance segmenters in resource-constrained clinical environments. Code: https://github.com/ClinicaAlpha/DPRD-3D-MedSeg
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）；Wuhan University（武汉大学）
Current Knowledge Distillation (KD) methods for semantic segmentation often guide the student to mimic the teacher's structured information generated from individual data samples. However, they ignore the global semantic relations among pixels across various images that are valuable for KD. This paper proposes a novel Cross-Image Relational KD (CIRKD), which focuses on transferring structured pixel-to-pixel and pixel-to-region relations among the whole images. The motivation is that a good teacher network could construct a well-structured feature space in terms of global pixel dependencies. CI…
Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conduct…
Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification
Victoria University（维多利亚大学）；Bangaldesh University of Business and Technology（孟加拉国商业与技术大学）；Daffodil Institute of Information Technology（达芙妮信息技术学院）；American International University-Bangladesh (AIUB)（孟加拉国美国国际大学）；Multimedia University（多媒体大学）
Airdoc LLC（鹰瞳科技）；Monash University（莫纳什大学）；Harbin Engineering University（哈尔滨工程大学）；The University of Sydney（悉尼大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy most of the data, while most classes have rarely few samples), which results in a challenging imbalance learning scenario. For example, there are estimated more than 40 different kinds of retinal diseases with variable morbidity, however with more than 30+ conditions are very rare from the global patient cohorts, which results in a typical long-tailed learning problem for deep learning-based screening models. In this study, we propose class subset learning by dividing the long-tailed d…
Nanyang Technological University（南洋理工大学）；University of British Columbia（不列颠哥伦比亚大学）；Vancouver Coast Health Research Institute（温哥华海岸健康研究所）；BC Cancer（不列颠哥伦比亚癌症中心）
Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Over the last few years, computer-aided diagnosis has been rapidly developed and make great progress in healthcare and medical practices due to the advances in artificial intelligence. However, most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. In this case, knowledge distillation (KD) has been proven as an efficient tool to help improve the adaptability o…
Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
Graduate School of Arts and Sciences, The University of Tokyo（东京大学文学部与科学研究科）；Graduate School of Engineering Science, The University of Osaka（大阪大学工程科学研究科）
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong（香港中文大学）；Xiamen University（厦门大学）；City University of Hong Kong（香港城市大学）
The amount of medical images for training deep classification models is typically very scarce, making these deep models prone to overfit the training data. Studies showed that knowledge distillation (KD), especially the mean-teacher framework which is more robust to perturbations, can help mitigate the over-fitting effect. However, directly transferring KD from computer vision to medical image classification yields inferior performance as medical images suffer from higher intra-class variance and class imbalance. To address these issues, we propose a novel Categorical Relation-preserving Contr…
On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及交叉应用重点实验室（东南大学））
Clemson University（克莱姆森大学）；University at Buffalo（布法罗大学）
Knowledge distillation (KD) has become an important technique for model compression and knowledge transfer. In this work, we first perform a comprehensive analysis of the knowledge transferred by different KD methods. We demonstrate that traditional KD methods, which minimize the KL divergence of softmax outputs between networks, are related to the knowledge alignment of an individual sample only. Meanwhile, recent contrastive learning-based KD methods mainly transfer relational knowledge between different samples, namely, knowledge correlation. While it is important to transfer the full knowl…
Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Adobe Research（奥多比研究院）；Arizona State University（亚利桑那州立大学）
For tasks involving language and vision, the current state-of-the-art methods tend not to leverage any additional information that might be present to gather relevant (commonsense) knowledge. A representative task is Visual Question Answering where large diagnostic datasets have been proposed to test a system's capability of answering questions about images. The training data is often accompanied by annotations of individual object properties and spatial locations. In this work, we take a step towards integrating this additional privileged information in the form of spatial knowledge to aid in…
CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）
Knowledge Distillation (KD) aims at transferring knowledge from a larger well-optimized teacher network to a smaller learnable student network.Existing KD methods have mainly considered two types of knowledge, namely the individual knowledge and the relational knowledge. However, these two types of knowledge are usually modeled independently while the inherent correlations between them are largely ignored. It is critical for sufficient student network learning to integrate both individual knowledge and relational knowledge while reserving their inherent correlation. In this paper, we propose t…
Local Dense Logit Relations for Enhanced Knowledge Distillation
Northeastern University, China（东北大学）；South China Normal University, China（华南师范大学）；Shenzhen Institutes of Advanced Technology, CAS, China（深圳先进技术研究院）；The Chinese University of Hong Kong, Hong Kong, China（香港中文大学）
Deep convolutional neural network with increased number of parameters has achieved improved precision in task of object detection on natural images, where objects of interests are annotated with horizontal boundary boxes. On aerial images captured from the bird-view perspective, these improvements on model architecture and deeper convolutional layers can also boost the performance on oriented object detection task. However, it is hard to directly apply those state-of-the-art object detectors on the devices with limited computation resources, which necessitates lightweight models through model …
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
Recent deep metric learning (DML) methods typically leverage solely class labels to keep positive samples far away from negative ones. However, this type of method normally ignores the crucial knowledge hidden in the data (e.g., intra-class information variation), which is harmful to the generalization of the trained model. To alleviate this problem, in this paper we propose Online Batch Diffusion-based Self-Distillation (OBD-SD) for DML. Specifically, we first propose a simple but effective Progressive Self-Distillation (PSD), which distills the knowledge progressively from the model itself d…
FreeKD: Knowledge Distillation via Semantic Frequency Prompt
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机学院）；The University of Sydney（悉尼大学）；Zhejiang University（浙江大学）
Quantum-Gated Task-interaction Knowledge Distillation for Pre-trained Model-based Class-Incremental Learning
School of Information and Communication Engineering, BUPT（北京邮电大学信息与通信工程学院）；BUPT: Beijing University of Posts and Telecommunications（北京邮电大学）；Engineering Research Center for Information Network, Ministry of Education（教育部信息网络工程研究中心）
Indian Institute of Technology, Jodhpur（焦特布尔印度理工学院）
Knowledge distillation is a common technique for improving the performance of a shallow student network by transferring information from a teacher network, which in general, is comparatively large and deep. These teacher networks are pre-trained and often uncalibrated, as no calibration technique is applied to the teacher model while training. Calibration of a network measures the probability of correctness for any of its predictions, which is critical in high-risk domains. In this paper, we study how to obtain a calibrated student from an uncalibrated teacher. Our approach relies on the fusio…
ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
SenseTime Research（商汤科技研究院）；The University of Sydney（悉尼大学）；University of Science and Technology of China（中国科学技术大学）
Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations be…
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Electrical and Information Engineering, The University of Sydney（悉尼大学电气与信息工程学院）
Scale-invariant and View-relational Representation Learning for Full Surround Monocular Depth
Department of Electrical Engineering & Computer Sciences, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（电子工程与计算机科学系，大邱庆北科学技术大学）；Department of Autonomous Driving Perception Technology Vanguard Team, Hyundai Motor Company（自动驾驶感知技术先锋团队，现代汽车公司）
