Multimodal learning that integrates genomics and histopathology has shown strong potential in cancer diagnosis, yet its clinical translation is hindered by the limited availability of paired histology-genomics data. Knowledge distillation (KD) offers a practical solution by transferring genomic supervision into histopathology models, enabling accurate inference using histology alone. However, existing KD methods rely on batch-local alignment, which introduces instability due to limited within-batch comparisons and ultimately degrades performance.
To address these limitations, we propose Momentum Memory Knowledge Distillation (MoMKD), a cross-modal distillation framework driven by a momentum-updated memory. This memory aggregates genomic and histopathology information across batches, effectively enlarging the supervisory context available to each mini-batch. Furthermore, we decouple the gradients of the genomics and histology branches, preventing genomic signals from dominating histology feature learning during training and eliminating the modality-gap issue at inference time.
Extensive experiments on the TCGA-BRCA benchmark (HER2, PR, and ODX classification tasks) and an independent in-house testing dataset demonstrate that MoMKD consistently outperforms state-of-the-art MIL and multimodal KD baselines, delivering strong performance and generalization under histology-only inference. Overall, MoMKD establishes a robust and generalizable knowledge distillation paradigm for computational pathology.
Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings
Technical University of Munich（慕尼黑技术大学）；Helmholtz Munich（马克斯·普朗克研究所慕尼黑分部）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
EKFZ for Digital Health（数字健康EKFZ）；TU Dresden（德累斯顿工业大学）；University of Augsburg（奥格斯堡大学）；Bavarian Cancer Research Center (BZKF)（巴伐利亚癌症研究中心）；Department of Medicine I, TU Dresden（德累斯顿工业大学第一医学院）；NCT Heidelberg（海德堡NCT）；University of Leeds（利兹大学）
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Washington（华盛顿大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Medical Optical Technology R&D Center, Research Institute of Tsinghua（清华研究院医学光学技术研发中心）；Jinfeng Laboratory（金凤实验室）
Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation
The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The First Affiliated Hospital of Shandong First Medical University and Shandong Provincial Qianfoshan Hospital（山东第一医科大学第一附属医院暨山东省千佛山医院）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
College of Computer Science, Sichuan University（四川大学计算机学院）；Sichuan University-Pittsburgh Institute（四川大学匹兹堡研究院）；Department of Pathology West China Hospital, Sichuan University（四川大学华西医院病理科部门）；Institute of Clinical Pathology, West China Hospital, Sichuan University（四川大学华西医院临床病理研究所）
Beijing University of Posts and Telecommunications（北京邮电大学）；Center for Research on Intelligent Perception and Computing, NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Shandong University of Science and Technology（山东科技大学）；Artificial Intelligence Research, CAS (CAS-AIR)（中国科学院人工智能研究院）
In recent years, deep convolutional neural networks have made significant advances in pathology image segmentation. However, pathology image segmentation encounters with a dilemma in which the higher-performance networks generally require more computational resources and storage. This phenomenon limits the employment of high-accuracy networks in real scenes due to the inherent high-resolution of pathological images. To tackle this problem, we propose CoCo DistillNet, a novel Cross-layer Correlation (CoCo) knowledge distillation network for pathological gastric cancer segmentation. Knowledge di…
Enhancing WSI-Based Survival Analysis with Report-Auxiliary Self-Distillation
School of Informatics, Xiamen University, Xiamen, China（厦门大学信息学院）；Shanghai Innovation Institution, Shanghai, China（上海创新机构）；Nanfang Hospital, Southern Medical University, Guangzhou, China（南方医科大学珠江医院）；School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei, China（中国科学技术大学人工智能与数据科学学院）；EuroMov Digital Health in Motion, Univ Montpellier, IMT Mines Ales, Ales, France（EuroMov数字健康运动）；Service de Médecine Nucléaire, Centre Hospitalier Universitaire de Nîmes, Université de Montpellier, Nîmes, France（蒙彼利埃大学尼姆大学医院核医学服务）
Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；College of Information Technology, United Arab Emirates University（阿联酋大学信息技术学院）
This paper tackles the problem of semi-supervised video object segmentation on resource-constrained devices, such as mobile phones. We formulate this problem as a distillation task, whereby we demonstrate that small space-time-memory networks with finite memory can achieve competitive results with state of the art, but at a fraction of the computational cost (32 milliseconds per frame on a Samsung Galaxy S22). Specifically, we provide a theoretically grounded framework that unifies knowledge distillation with supervised contrastive representation learning. These models are able to jointly bene…
Computer-aided pathology diagnosis based on the classification of Whole Slide Image (WSI) plays an important role in clinical practice, and it is often formulated as a weakly-supervised Multiple Instance Learning (MIL) problem. Existing methods solve this problem from either a bag classification or an instance classification perspective. In this paper, we propose an end-to-end weakly supervised knowledge distillation framework (WENO) for WSI classification, which integrates a bag classifier and an instance classifier in a knowledge distillation framework to mutually improve the performance of …
University of Maryland（马里兰大学）；Adobe Research（奥多比研究中心）
Incremental learning targets at achieving good performance on new categories without forgetting old ones. Knowledge distillation has been shown critical in preserving the performance on old classes. Conventional methods, however, sequentially distill knowledge only from the last model, leading to performance degradation on the old classes in later incremental learning steps. In this paper, we propose a multi-model and multi-level knowledge distillation strategy. Instead of sequentially distilling knowledge only from the last model, we directly leverage all previous model snapshots. In addition…
Dataset Distillation via Adversarial Prediction Matching
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61所）；University of Electronic Science and Technology of China（电子科技大学）；Dongguan University of Technology（东莞理工学院）
Xiamen University（厦门大学）；National University of Singapore（新加坡国立大学）
The large memory and computation consumption in convolutional neural networks (CNNs) has been one of the main barriers for deploying them on resource-limited systems. To this end, most cheap convolutions (e.g., group convolution, depth-wise convolution, and shift convolution) have recently been used for memory and computation reduction but with the specific architecture designing. Furthermore, it results in a low discriminability of the compressed networks by directly replacing the standard convolution with these cheap ones. In this paper, we propose to use knowledge distillation to improve th…
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University（江苏大学）；Griffith University（格里菲斯大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Fujian University of Technology（福建理工大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国学院）；Queen Mary University of London（伦敦玛丽女王大学）
Beihang University（北京航空航天大学）；iQIYI, Inc（爱奇艺公司）；Peking University（北京大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Large convolutional neural network models have recently demonstrated impressive performance on video attention prediction. Conventionally, these models are with intensive computation and large memory. To address these issues, we design an extremely light-weight network with ultrafast speed, named UVA-Net. The network is constructed based on depth-wise convolutions and takes low-resolution images as input. However, this straight-forward acceleration method will decrease performance dramatically. To this end, we propose a coupled knowledge distillation strategy to augment and train the network e…
TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
CUNY City College of NY（纽约城市大学城市学院）；Wyze Labs, Inc.（威智实验室公司）；University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）
Queen Mary University of London（伦敦玛丽女王大学）；Vision Semantics Ltd（视觉语义有限公司）
Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultan…
Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generation
UNSW Sydney（新南威尔士大学悉尼分校）；University of Technology Sydney（技术大学悉尼分校）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Stanford University（斯坦福大学）；Shanghai Jiao Tong University（上海交通大学）
Harvard Medical School（哈佛医学院）；University of Toronto（多伦多大学）
Tissue phenotyping is a fundamental task in learning objective characterizations of histopathologic biomarkers within the tumor-immune microenvironment in cancer pathology. However, whole-slide imaging (WSI) is a complex computer vision in which: 1) WSIs have enormous image resolutions with precludes large-scale pixel-level efforts in data curation, and 2) diversity of morphological phenotypes results in inter- and intra-observer variability in tissue labeling. To address these limitations, current efforts have proposed using pretrained image encoders (transfer learning from ImageNet, self-sup…
Harnessing Adversarial Distillation to Customise Debiased, Disease-Specific Pathology Foundation Models for Breast Cancer
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Computing and Mathematical Sciences, University of Leicester（莱斯特大学计算与数学科学学院）；Leicester Cancer Research Centre, University of Leicester（莱斯特大学莱斯特癌症研究中心）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Nuffield Department of Medicine, University of Oxford（牛津大学纳菲尔德医学院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）；ZoyMed（佐医医疗（ZoyMed））
University of Trento（特伦托大学）；SAP AI Research（SAP人工智能研究院）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信LTCI实验室）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Continual Learning (CL) aims to develop agents emulating the human ability to sequentially learn new tasks while being able to retain knowledge obtained from past experiences. In this paper, we introduce the novel problem of Memory-Constrained Online Continual Learning (MC-OCL) which imposes strict constraints on the memory overhead that a possible algorithm can use to avoid catastrophic forgetting. As most, if not all, previous CL methods violate these constraints, we propose an algorithmic solution to MC-OCL: Batch-level Distillation (BLD), a regularization-based CL approach, which effective…
Unifying Multiple Foundation Models for Advanced Computational Pathology
Shanghai Jiao Tong University（上海交通大学）；Washington University in St. Louis（华盛顿大学）；University of Science and Technology Beijing（北京科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Department of Surgical Oncology and General Surgery, Key Laboratory of Precision Diagnosis and Treatment of Gastrointestinal Tumours, Ministry of Education, The First Hospital of China Medical University（外科肿瘤科和普通外科，国家教育委员会胃肠道肿瘤精准诊断与治疗重点实验室，中国医科大学第一医院）；Shanghai Innovation Institute（上海创新研究院）；Sensetime Research（商汤科技研究院）
University of Toronto（多伦多大学）；Toronto AI Lab（多伦多人工智能实验室）；LG Electronics Canada（LG电子加拿大公司）
Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achiev…
Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footpri…
TriDeNT: Triple Deep Network Training for Privileged Knowledge Distillation in Histopathology
School of Cancer Sciences, University of Glasgow（格拉斯哥大学癌症科学学院）；Cancer Research UK Scotland Institute（英国癌症研究苏格兰研究所）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学院）；Division of Biosciences, University College London（伦敦大学学院生物科学部）
