Segmenting tumors in histological images is vital for cancer diagnosis. While fully supervised models excel with pixel-level annotations, creating such annotations is labor-intensive and costly. Accurate histopathology image segmentation under weakly-supervised conditions with coarse-grained image labels is still a challenging problem. Although multiple instance learning (MIL) has shown promise in segmentation tasks, surprisingly, no previous pseudo-supervision methods have used MIL-based outputs as pseudo-masks for training. We suspect this stems from concerns over noises in MIL results affecting pseudo supervision quality. To explore the potential of leveraging MIL-based segmentation for pseudo supervision, we propose a novel distillation framework for histopathology image segmentation. This framework introduces a iterative fusion-knowledge distillation strategy, enabling the student model to learn directly from the teacher's comprehensive outcomes. Through dynamic role reversal between the fixed teacher and learnable student models and the incorporation of weighted cross-entropy loss for model optimization, our approach prevents performance deterioration and noise amplification during knowledge distillation. Experimental results on public histopathology datasets, Camelyon16 and Digestpath2019, demonstrate that our approach not only complements various MIL-based segmentation methods but also significantly enhances their performance. Additionally, our method achieves new SOTA in the field.
Dataset Distillation for Histopathology Image Classification
Australian Institute of Health Innovation (AIHI), Macquarie University（澳大利亚麦考瑞大学澳大利亚健康创新研究所）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, Peking University（北京大学计算机学院）；School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）
TriDeNT: Triple Deep Network Training for Privileged Knowledge Distillation in Histopathology
School of Cancer Sciences, University of Glasgow（格拉斯哥大学癌症科学学院）；Cancer Research UK Scotland Institute（英国癌症研究苏格兰研究所）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学院）；Division of Biosciences, University College London（伦敦大学学院生物科学部）
Harvard Medical School（哈佛医学院）；University of Toronto（多伦多大学）
Tissue phenotyping is a fundamental task in learning objective characterizations of histopathologic biomarkers within the tumor-immune microenvironment in cancer pathology. However, whole-slide imaging (WSI) is a complex computer vision in which: 1) WSIs have enormous image resolutions with precludes large-scale pixel-level efforts in data curation, and 2) diversity of morphological phenotypes results in inter- and intra-observer variability in tissue labeling. To address these limitations, current efforts have proposed using pretrained image encoders (transfer learning from ImageNet, self-sup…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；IBM Research（IBM研究院）
Weakly supervised instance labeling using only image-level labels, in lieu of expensive fine-grained pixel annotations, is crucial in several applications including medical image analysis. In contrast to conventional instance segmentation scenarios in computer vision, the problems that we consider are characterized by a small number of training images and non-local patterns that lead to the diagnosis. In this paper, we explore the use of multiple instance learning (MIL) to design an instance label generator under this weakly supervised setting. Motivated by the observation that an MIL model ca…
University of Liverpool（利物浦大学）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）；China Science IntelliCloud Technology Co., Ltd（中科智云科技有限公司）；Remark AI UK Limited（Remark AI英国有限公司）
Multiple instance learning (MIL) has been increasingly used in the classification of histopathology whole slide images (WSIs). However, MIL approaches for this specific classification problem still face unique challenges, particularly those related to small sample cohorts. In these, there are limited number of WSI slides (bags), while the resolution of a single WSI is huge, which leads to a large number of patches (instances) cropped from this slide. To address this issue, we propose to virtually enlarge the number of bags by introducing the concept of pseudo-bags, on which a double-tier MIL f…
Multi-modal Knowledge Decomposition based Online Distillation for Biomarker Prediction in Breast Cancer Histopathology
School of Biomedical Engineering, Faulty of Medicine, Dalian University of Technology, Dalian, China（生物医学工程学院）；Faculty of Information Technology, University of Jyvaskyla, Jyvaskyla, Finland（信息科技学院）；School of Software Technology, Dalian University of Technology, Dalian, China（软件技术学院）；Department of Radiology, Guangdong Provincial People’s Hospital, Southern Medical University, Guangzhou, China（放射科）
Breast cancer is the most commonly diagnosed cancer worldwide, with over two million new cases each year. During diagnostic tumour grading, pathologists manually count the number of dividing cells (mitotic figures) in biopsy or tumour resection specimens. Since the process is subjective and time-consuming, data-driven artificial intelligence (AI) methods have been developed to automatically detect mitotic figures. However, these methods often generalise poorly, with performance reduced by variations in tissue types, staining protocols, or the scanners used to digitise whole-slide images. Domai…
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Machine Learning Group, Technische Universität Berlin（柏林工业大学机器学习组）；Aignostics GmbH（艾格诺斯蒂克斯有限公司）；Institute of Pathology, Ludwig Maximilian University（路德维希-马克西米利安大学病理学研究所）；German Cancer Research Center（德国癌症研究中心）；Institute of Pathology, Charité Universitätsmedizin（夏里特医学院病理学研究所）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Washington（华盛顿大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Medical Optical Technology R&D Center, Research Institute of Tsinghua（清华研究院医学光学技术研发中心）；Jinfeng Laboratory（金凤实验室）
ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation
Ho Chi Minh City University of Technology（胡志明市技术大学）；University of Technology Sydney（悉尼技术大学）；University of Houston（休斯顿大学）；University of Information Technology（信息科技大学）；College of Medical Science and Technology, Taipei Medical University（台北医学院医学科技学院）；Department of Computer Science, Emory University（埃默里大学计算机科学系）；Montefiore Medical Center, Albert Einstein College of Medicine（蒙特福伊医疗中心，阿尔伯特·爱因斯坦医学院）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）
Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
College of Computer Science, Sichuan University（四川大学计算机学院）；Sichuan University-Pittsburgh Institute（四川大学匹兹堡研究院）；Department of Pathology West China Hospital, Sichuan University（四川大学华西医院病理科部门）；Institute of Clinical Pathology, West China Hospital, Sichuan University（四川大学华西医院临床病理研究所）
This paper tackles the problem of semi-supervised video object segmentation on resource-constrained devices, such as mobile phones. We formulate this problem as a distillation task, whereby we demonstrate that small space-time-memory networks with finite memory can achieve competitive results with state of the art, but at a fraction of the computational cost (32 milliseconds per frame on a Samsung Galaxy S22). Specifically, we provide a theoretically grounded framework that unifies knowledge distillation with supervised contrastive representation learning. These models are able to jointly bene…
Fine-tuning a Multiple Instance Learning Feature Extractor with Masked Context Modelling and Knowledge Distillation
Institute for Biomedical Informatics, Faculty of Medicine（医学院生物医学信息学研究所）；University Hospital Cologne, University of Cologne（科隆大学医院（科隆大学））；Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine（科隆分子医学中心（CMMC，医学院））
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of California, Berkeley（加州大学伯克利分校）
Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequen…
SlideGCD: Slide-based Graph Collaborative Training with Knowledge Distillation for Whole Slide Image Classification
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；School of Software, Hefei University of Technology（合肥工业大学软件学院）；Image Processing Center, School of Astronautics, Beihang University（北京航空航天大学宇航学院图像处理中心）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, …
University of Science and Technology of China（中国科学技术大学）；National Engineering Laboratory for Speech and Language Information Processing（语音及语言信息处理国家工程实验室）
The technique of distillation helps transform cumbersome neural network into compact network so that the model can be deployed on alternative hardware devices. The main advantages of distillation based approaches include simple training process, supported by most off-the-shelf deep learning softwares and no special requirement of hardwares. In this paper, we propose a guideline to distill the architecture and knowledge of pre-trained standard CNNs simultaneously. We first make a quantitative analysis of the baseline network, including computational cost and storage overhead in different compon…
LadderMIL: Multiple Instance Learning with Coarse-to-Fine Self-Distillation
Institute for Regeneration and Repair, University of Edinburgh（再生与修复研究所，爱丁堡大学）；School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；Indica Labs（Indica实验室）；School of Medicine, University of St Andrews（医学学院，圣安德鲁斯大学）；Department of Basic Pathology, National Defense Medical College（基础病理学部门，国家防卫医疗学院）
Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models
School of Basic Medical Sciences, Fudan University（复旦大学基础医学院）；Shanghai Key Laboratory of MICCAI（上海市MICCAI重点实验室）；Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究所）；Shanghai Collaborative Innovation Centre of Endoscopy（上海市内镜微创技术协同创新中心）；Endoscopy Centre and Endoscopy Research Institute, Zhongshan Hospital, Fudan University（复旦大学附属中山医院内镜中心及内镜研究所）；Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）
Computer-aided pathology diagnosis based on the classification of Whole Slide Image (WSI) plays an important role in clinical practice, and it is often formulated as a weakly-supervised Multiple Instance Learning (MIL) problem. Existing methods solve this problem from either a bag classification or an instance classification perspective. In this paper, we propose an end-to-end weakly supervised knowledge distillation framework (WENO) for WSI classification, which integrates a bag classifier and an instance classifier in a knowledge distillation framework to mutually improve the performance of …
With the improvement of AI chips (e.g., GPU, TPU, and NPU) and the fast development of the Internet of Things (IoT), some robust deep neural networks (DNNs) are usually composed of millions or even hundreds of millions of parameters. Such a large model may not be suitable for directly deploying on low computation and low capacity units (e.g., edge devices). Knowledge distillation (KD) has recently been recognized as a powerful model compression method to decrease the model parameters effectively. The central concept of KD is to extract useful information from the feature maps of a large model …
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Cross-modal encoders for vision-language (VL) tasks are often pretrained with carefully curated vision-language datasets. While these datasets reach an order of 10 million samples, the labor cost is prohibitive to scale further. Conversely, unimodal encoders are pretrained with simpler annotations that are less cost-prohibitive, achieving scales of hundreds of millions to billions. As a result, unimodal encoders have achieved state-of-art (SOTA) on many downstream tasks. However, challenges remain when applying to VL tasks. The pretraining data is not optimal for cross-modal architectures and …
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Microsoft Research（微软研究院）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
In this paper, we develop a new weakly-supervised learning algorithm to learn to segment cancerous regions in histopathology images. Our work is under a multiple instance learning framework (MIL) with a new formulation, deep weak supervision (DWS); we also propose an effective way to introduce constraints to our neural networks to assist the learning process. The contributions of our algorithm are threefold: (1) We build an end-to-end learning system that segments cancerous regions with fully convolutional networks (FCN) in which image-to-image weakly-supervised learning is performed. (2) We d…
Knowledge Distillation for mmWave Beam Prediction Using Sub-6 GHz Channels
Centre for Wireless Communications, University of Oulu, P.O.Box 4500, FI-90014, Finland（奥卢大学无线通信中心，芬兰）；School of Electrical, Computer, and Energy Engineering, Arizona State University, AZ, USA（亚利桑那州立大学电气、计算机与能源工程学院）
Hispathological image segmentation algorithms play a critical role in computer aided diagnosis technology. The development of weakly supervised segmentation algorithm alleviates the problem of medical image annotation that it is time-consuming and labor-intensive. As a subset of weakly supervised learning, Multiple Instance Learning (MIL) has been proven to be effective in segmentation. However, there is a lack of related information between instances in MIL, which limits the further improvement of segmentation performance. In this paper, we propose a novel weakly supervised method for pixel-l…
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）
Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN lea…
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
