This paper addresses complex challenges in histopathological image analysis through three key contributions. Firstly, it introduces a fast patch selection method, FPS, for whole-slide image (WSI) analysis, significantly reducing computational cost while maintaining accuracy. Secondly, it presents PathDino, a lightweight histopathology feature extractor with a minimal configuration of five Transformer blocks and only 9 million parameters, markedly fewer than alternatives. Thirdly, it introduces a rotation-agnostic representation learning paradigm using self-supervised learning, effectively mitigating overfitting. We also show that our compact model outperforms existing state-of-the-art histopathology-specific vision transformers on 12 diverse datasets, including both internal datasets spanning four sites (breast, liver, skin, and colorectal) and seven public datasets (PANDA, CAMELYON16, BRACS, DigestPath, Kather, PanNuke, and WSSS4LUAD). Notably, even with a training dataset of 6 million histopathology patches from The Cancer Genome Atlas (TCGA), our approach demonstrates an average 8.5% improvement in patch-level majority vote performance. These contributions provide a robust framework for enhancing image analysis in digital pathology, rigorously validated through extensive evaluation. Project Page: https://kimialabmayo.github.io/PathDino-Page/
SagivTech Ltd.（SagivTech有限公司）；Roche Innovation Center Munich（慕尼黑罗氏创新中心）
Using features extracted from networks pretrained on ImageNet is a common practice in applications of deep learning for digital pathology. However it presents the downside of missing domain specific image information. In digital pathology, supervised training data is expensive and difficult to collect. We propose a self-supervised method for feature extraction by similarity learning on whole slide images (WSI) that is simple to implement and allows creation of robust and compact image descriptors. We train a siamese network, exploiting image spatial continuity and assuming spatially adjacent t…
Kharkevich Institute for Information Transmission Problems（信息传输问题研究所（Kharkevich研究所））；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
In the last years, neural networks have proven to be a powerful framework for various image analysis problems. However, some application domains have specific limitations. Notably, digital pathology is an example of such fields due to tremendous image sizes and quite limited number of training examples available. In this paper, we adopt state-of-the-art convolutional neural networks (CNN) architectures for digital pathology images analysis. We propose to classify image patches to increase effective sample size and then to apply an ensembling technique to build prediction for the original image…
University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）
In Digital Pathology (DP), labeled data is generally very scarce due to the requirement that medical experts provide annotations. We address this issue by learning transferable features from weakly labeled data, which are collected from various parts of the body and are organized by non-medical experts. In this paper, we show that features learned from such weakly labeled datasets are indeed transferable and allow us to achieve highly competitive patch classification results on the colorectal cancer (CRC) dataset [1] and the PatchCamelyon (PCam) dataset [2] while using an order of magnitude le…
EPFL（洛桑联邦理工学院）；IBM Research Zürich（IBM苏黎世研究院）；ETH Zürich（苏黎世联邦理工学院）；National Cancer Institute - IRCCS-Fondazione Pascale（国家癌症研究所 - IRCCS帕斯卡尔基金会）
Explainability of machine learning (ML) techniques in digital pathology (DP) is of great significance to facilitate their wide adoption in clinics. Recently, graph techniques encoding relevant biological entities have been employed to represent and assess DP images. Such paradigm shift from pixel-wise to entity-wise analysis provides more control over concept representation. In this paper, we introduce a post-hoc explainer to derive compact per-instance explanations emphasizing diagnostically important entities in the graph. Although we focus our analyses to cells and cellular interactions in …
Explaining Digital Pathology Models via Clustering Activations
Faculty of Informatics, Masaryk University, Brno, Czech Republic（马萨里克大学信息学院，布拉格，捷克共和国）；Institute of Computer Science, Masaryk University, Brno, Czech Republic（马萨里克大学计算机科学研究所，布拉格，捷克共和国）；Masaryk Memorial Cancer Institute, Brno, Czech Republic（马萨里克纪念癌症研究所，布拉格，捷克共和国）
Discrepancy Minimization Improves Cross-Hospital Robustness in Digital Pathology
Reichman University, Herzliya, Israel（以色列海法大学）；Institute of Pathology, Sheba Medical Center, Ramat-Gan, Israel（以色列沙巴医疗中心病理研究所）；Technion - Israel Institute of Technology, Haifa, Israel（以色列技术学院）
Normalizing unwanted color variations due to differences in staining processes and scanner responses has been shown to aid machine learning in computational pathology. Of the several popular techniques for color normalization, structure preserving color normalization (SPCN) is well-motivated, convincingly tested, and published with its code base. However, SPCN makes occasional errors in color basis estimation leading to artifacts such as swapping the color basis vectors between stains or giving a colored tinge to the background with no tissue. We made several algorithmic improvements to remove…
Jitendra Malik once said, "Supervision is the opium of the AI researcher". Most deep learning techniques heavily rely on extreme amounts of human labels to work effectively. In today's world, the rate of data creation greatly surpasses the rate of data annotation. Full reliance on human annotations is just a temporary means to solve current closed problems in AI. In reality, only a tiny fraction of data is annotated. Annotation Efficient Learning (AEL) is a study of algorithms to train models effectively with fewer annotations. To thrive in AEL environments, we need deep learning techniques th…
Stony Brook University（石溪大学）；Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Fraunhofer Institute for Medical Image Computing MEVIS（弗劳恩霍夫医学图像计算研究所）；University of Bremen（不来梅大学）
Histopathological assessments, including surgical resection and core needle biopsy, are the standard procedures in the diagnosis of the prostate cancer. Current interpretation of the histopathology images includes the determination of the tumor area, Gleason grading, and identification of certain prognosis-critical features. Such a process is not only tedious, but also prune to intra/inter-observe variabilities. Recently, FDA cleared the marketing of the first whole slide imaging system for digital pathology. This opens a new era for the computer aided prostate image analysis and feature extra…
University of Dayton（代顿大学）；Deep Lens Inc（Deep Lens公司）
Deep Learning (DL) approaches have been providing state-of-the-art performance in different modalities in the field of medical imagining including Digital Pathology Image Analysis (DPIA). Out of many different DL approaches, Deep Convolutional Neural Network (DCNN) technique provides superior performance for classification, segmentation, and detection tasks. Most of the task in DPIA problems are somehow possible to solve with classification, segmentation, and detection approaches. In addition, sometimes pre and post-processing methods are applied for solving some specific type of problems. Rec…
Roche Innovation Center Munich（慕尼黑罗氏创新中心）；DeePathology Ltd.（DeePathology有限公司）
Histopathological evaluation of tissue samples is a key practice in patient diagnosis and drug development, especially in oncology. Historically, Hematoxylin and Eosin (H&E) has been used by pathologists as a gold standard staining. However, in many cases, various target specific stains, including immunohistochemistry (IHC), are needed in order to highlight specific structures in the tissue. As tissue is scarce and staining procedures are tedious, it would be beneficial to generate images of stained tissue virtually. Virtual staining could also generate in-silico multiplexing of different stai…
Foundation Models for Slide-level Cancer Subtyping in Digital Pathology
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-Tech)（以人为本技术大学研究院（HUMAN-Tech））；Universitat Politécnica de Valencia（瓦伦西亚理工大学）；Valencian Graduate School and Research Network for Artificial Intelligence (valgraAI)（瓦伦西亚人工智能研究生院与研究网络（valgraAI））
University of Toronto（多伦多大学）；Huron Digital Pathology Inc.（休伦数字病理公司）
One of the challenges facing the adoption of digital pathology workflows for clinical use is the need for automated quality control. As the scanners sometimes determine focus inaccurately, the resultant image blur deteriorates the scanned slide to the point of being unusable. Also, the scanned slide images tend to be extremely large when scanned at greater or equal 20X image resolution. Hence, for digital pathology to be clinically useful, it is necessary to use computational tools to quickly and accurately quantify the image focus quality and determine whether an image needs to be re-scanned.…
Quantitative Benchmarking of Anomaly Detection Methods in Digital Pathology
Department of Computer Science Vanderbilt University, TN, USA（计算机科学系范德堡大学）；Department of Radiology Weill Cornell Medicine, NY, USA（放射学系韦尔·科恩医学中心）；Department of Medicine Vanderbilt University Medical Center, TN, USA Veterans Affairs Tennessee Valley Healthcare System TN, USA（医学系范德堡大学医学中心退伍军人事务田纳西山谷医疗系统）；Department of Medicine Vanderbilt University Medical Center Veterans Affairs Tennessee Valley Healthcare System TN, USA（医学系范德堡大学医学中心退伍军人事务田纳西山谷医疗系统）；Department of Anatomical Pathology UT MD Anderson Cancer Center,TX, USA（解剖病理学系UT MD安德森癌症中心）
Linköping University（林雪平大学）；Center for Medical Image Science and Visualization, Linköping University（林雪平大学医学影像科学与可视化中心）；Sectra AB
Artificial intelligence (AI) has shown great promise for diagnostic imaging assessments. However, the application of AI to support medical diagnostics in clinical routine comes with many challenges. The algorithms should have high prediction accuracy but also be transparent, understandable and reliable. Thus, explainable artificial intelligence (XAI) is highly relevant for this domain. We present a survey on XAI within digital pathology, a medical imaging sub-discipline with particular characteristics and needs. The review includes several contributions. Firstly, we give a thorough overview of…
Barco Healthcare（巴可医疗）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
It is our conjecture that the variability of colors in a pathology image effects the interpretation of pathology cases, whether it is diagnostic accuracy, diagnostic confidence, or workflow efficiency. In this paper, digital pathology images are analyzed to quantify the perceived difference in color that occurs due to display aging, in particular a change in the maximum luminance, white point, and color gamut. The digital pathology images studied include diagnostically important features, such as the conspicuity of nuclei. Three different display aging models are applied to images: aging of lu…
University of Guelph（圭尔夫大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
We consider annotation efficient learning in Digital Pathology (DP), where expert annotations are expensive and thus scarce. We explore the impact of image and input resolution on DP patch classification performance. We use two cancer patch classification datasets PCam and CRC, to validate the results of our study. Our experiments show that patch classification performance can be improved by manipulating both the image and input resolution in annotation-scarce and annotation-rich environments. We show a positive correlation between the image and input resolution and the patch classification ac…
IBM Research（IBM研究院）；EPFL（洛桑联邦理工学院）；ETH（苏黎世联邦理工学院）
Advances in entity-graph based analysis of histopathology images have brought in a new paradigm to describe tissue composition, and learn the tissue structure-to-function relationship. Entity-graphs offer flexible and scalable representations to characterize tissue organization, while allowing the incorporation of prior pathological knowledge to further support model interpretability and explainability. However, entity-graph analysis requires prerequisites for image-to-graph translation and knowledge of state-of-the-art machine learning algorithms applied to graph-structured data, which can po…
Machine learning (ML) algorithms are optimized for the distribution represented by the training data. For outlier data, they often deliver predictions with equal confidence, even though these should not be trusted. In order to deploy ML-based digital pathology solutions in clinical practice, effective methods for detecting anomalous data are crucial to avoid incorrect decisions in the outlier scenario. We propose a new unsupervised learning approach for anomaly detection in histopathology data based on generative adversarial networks (GANs). Compared to the existing GAN-based methods that have…
University of Waterloo（滑铁卢大学）；Ontario Tech University（安大略理工大学）；Mayo Clinic（梅奥诊所）
Whole Slide Images (WSIs) in digital pathology are used to diagnose cancer subtypes. The difference in procedures to acquire WSIs at various trial sites gives rise to variability in the histopathology images, thus making consistent diagnosis challenging. These differences may stem from variability in image acquisition through multi-vendor scanners, variable acquisition parameters, and differences in staining procedure; as well, patient demographics may bias the glass slide batches before image acquisition. These variabilities are assumed to cause a domain shift in the images of different hospi…
Unlocking adaptive digital pathology through dynamic feature learning
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Oxford（牛津大学）；Fourth Military Medical University（第四军医大学）；Third Military Medical University（第三军医大学）；Chongqing Advanced Pathology Research Institute（重庆先进病理研究院）；Jinfeng Laboratory（金凤实验室）；Soochow University（苏州大学）；The Affiliated Cancer Hospital of Zhengzhou University and Henan Cancer Hospital（郑州大学附属肿瘤医院暨河南省肿瘤医院）；Research Institute of Tsinghua, Pearl River Delta（清华大学珠三角研究院）
In this work, we investigate multi-task learning as a way of pre-training models for classification tasks in digital pathology. It is motivated by the fact that many small and medium-size datasets have been released by the community over the years whereas there is no large scale dataset similar to ImageNet in the domain. We first assemble and transform many digital pathology datasets into a pool of 22 classification tasks and almost 900k images. Then, we propose a simple architecture and training scheme for creating a transferable model and a robust evaluation and selection protocol in order t…
We propose a new model for digital pathology segmentation, based on the observation that histopathology images are inherently symmetric under rotation and reflection. Utilizing recent findings on rotation equivariant CNNs, the proposed model leverages these symmetries in a principled manner. We present a visual analysis showing improved stability on predictions, and demonstrate that exploiting rotation equivariance significantly improves tumor detection performance on a challenging lymph node metastases dataset. We further present a novel derived dataset to enable principled comparison of mach…
Stony Brook University（石溪大学）；Snap Inc.（Snap公司）；CentraleSupélec（中央理工-高等电力学院）；University of Paris-Saclay（巴黎萨克雷大学）
Dense prediction tasks such as segmentation and detection of pathological entities hold crucial clinical value in computational pathology workflows. However, obtaining dense annotations on large cohorts is usually tedious and expensive. Contrastive learning (CL) is thus often employed to leverage large volumes of unlabeled data to pre-train the backbone network. To boost CL for dense prediction, some studies have proposed variations of dense matching objectives in pre-training. However, our analysis shows that employing existing dense matching strategies on histopathology images enforces invar…
Nanjing University of Posts and Telecommunications（南京邮电大学）；Nanjing University of Science and Technology（南京理工大学）；Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education（高维信息智能感知与系统教育部重点实验室）
In this paper, we propose an easily trained yet powerful representation learning approach with performance highly competitive to deep neural networks in a digital pathology image segmentation task. The method, called sparse coding driven deep decision tree ensembles that we abbreviate as ScD2TE, provides a new perspective on representation learning. We explore the possibility of stacking several layers based on non-differentiable pairwise modules and generate a densely concatenated architecture holding the characteristics of feature map reuse and end-to-end dense learning. Under this architect…
Deep learning (DL) has shown great potential in digital pathology applications. The robustness of a diagnostic DL-based solution is essential for safe clinical deployment. In this work we evaluate if adding uncertainty estimates for DL predictions in digital pathology could result in increased value for the clinical applications, by boosting the general predictive performance or by detecting mispredictions. We compare the effectiveness of model-integrated methods (MC dropout and Deep ensembles) with a model-agnostic approach (Test time augmentation, TTA). Moreover, four uncertainty metrics are…
Sunnybrook Research Institute（桑尼布鲁克研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
To alleviate the burden of gathering detailed expert annotations when training deep neural networks, we propose a weakly supervised learning approach to recognize metastases in microscopic images of breast lymph nodes. We describe an alternative training loss which clusters weakly labeled bags in latent space to inform relevance of patch-instances during training of a convolutional neural network. We evaluate our method on the Camelyon dataset which contains high-resolution digital slides of breast lymph nodes, where labels are provided at the image-level and only subsets of patches are made a…
Comparing ImageNet Pre-training with Digital Pathology Foundation Models for Whole Slide Image-Based Survival Analysis
Imperial College London, Department of Electrical and Electronic Engineering, London, UK（帝国理工学院伦敦分校，电子与电气工程系）；Department of Pathology, Institut de Pathologie Multisite, Groupement Hospitalier Sud, Lyon University Hospital, Pierre-Bénite, France（病理学系，多站点病理研究所，南部医院集团，里昂大学医院，皮埃尔-贝内蒂，法国）；University of Lyon, Université Claude Bernard Lyon 1, Lyon, France（里昂大学， Claude Bernard 里昂第一大学，里昂，法国）
Renal digital pathology visual knowledge search platform based on language large model and book knowledge
College of Information Technology, ZheJiang Shuren University（浙江树人大学信息技术学院）；The First Affiliated Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属第一医院）；Chinese Academy of Medical Science（中国医学科学院）；Alibaba-Zhejiang University Joint Research Center of Future Digital Healthcare（阿里巴巴-浙江大学未来数字医疗联合研究中心）；Zhejiang University School of Medicine（浙江大学医学院）
When designing a diagnostic model for a clinical application, it is crucial to guarantee the robustness of the model with respect to a wide range of image corruptions. Herein, an easy-to-use benchmark is established to evaluate how deep neural networks perform on corrupted pathology images. Specifically, corrupted images are generated by injecting nine types of common corruptions into validation images. Besides, two classification and one ranking metrics are designed to evaluate the prediction and confidence performance under corruption. Evaluated on two resulting benchmark datasets, we find t…
Revisiting Automatic Data Curation for Vision Foundation Models in Digital Pathology
Dept. of Computer Science, ETH Zurich, Zurich, Switzerland（苏黎世联邦理工学院计算机科学系）；AI Center, ETH Zurich, Zurich, Switzerland（苏黎世联邦理工学院人工智能中心）；Signal Processing Laboratory (LTS4), EPFL, Lausanne, Switzerland（日内瓦联邦理工学院信号处理实验室）；University of Basel, Basel, Switzerland（巴塞尔大学）；University Hospital of Basel, Basel, Switzerland（巴塞尔大学医院）；Idiap Research Institute, Martigny, Switzerland（伊迪普研究 institute）；Icahn School of Medicine at Mount Sinai, New York, United States（辛辛那提医学中心伊坎医学院）
This project aims to break down large pathology images into small tiles and then cluster those tiles into distinct groups without the knowledge of true labels, our analysis shows how difficult certain aspects of clustering tumorous and non-tumorous cells can be and also shows that comparing the results of different unsupervised approaches is not a trivial task. The project also provides a software package to be used by the digital pathology community, that uses some of the approaches developed to perform unsupervised unsupervised tile classification, which could then be easily manually labelle…
Deep Learning-Based Fixation Type Prediction for Quality Assurance in Digital Pathology
Institute of Pathology, Technical University of Munich, Germany（慕尼黑技术大学病理学研究所）；School of Computation, Information and Technology, Technical University of Munich, Germany（慕尼黑技术大学计算、信息与技术学院）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；Institute of Pathology, University of Regensburg, Regensburg, Germany（罗森堡大学病理学研究所）；Institute of Pathology and Molecular Diagnostics, University Hospital of Augsburg, Germany（埃格利希大学医院病理学与分子诊断研究所）；Digital Medicine, University Hospital of Augsburg, Germany（埃格利希大学医院数字医学）；Bavarian Cancer Research Center (BZKF), Augsburg, Germany（巴伐利亚癌症研究中心（BZKF））；Bio M {}^{\text{M}} Biotech Cluster Development GmbH, Martinsried, Germany（Bio M {}^{\text{M}} 生物技术集群发展 GmbH）；Institute of Pathology, Medical Faculty Mannheim, University Heidelberg, Germany（海德堡大学曼海姆医学学院病理学研究所）；Munich Data Science Institute, Technical University of Munich, Germany（慕尼黑数据科学研究所）
University of Science and Technology of China（中国科学技术大学）；UNSW Sydney（新南威尔士大学）；Genmab Inc.（Genmab公司）
NLP-based computer vision models, particularly vision transformers, have been shown to outperform CNN models in many imaging tasks. However, most digital pathology artificial-intelligence models are based on CNN architectures, probably owing to a lack of data regarding NLP models for pathology images. In this study, we developed digital pathology pipelines to benchmark the five most recently proposed NLP models (vision transformer (ViT), Swin Transformer, MobileViT, CMT, and Sequencer2D) and four popular CNN models (ResNet18, ResNet50, MobileNetV2, and EfficientNet) to predict biomarkers in co…
The impact of tissue detection on diagnostic artificial intelligence algorithms in digital pathology
Karolinska Institutet（卡罗林斯卡学院）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）；University of Bergen（卑尔根大学）；Synlab；Medical University of Lodz（罗兹医科大学）；University of Helsinki（赫尔辛基大学）；Mehiläinen Oy（梅希莱宁公司）；Mehiläinen Länsi-Pohja Hospital（梅希莱宁兰西波赫亚医院）；Aarhus University Hospital（奥胡斯大学医院）；Aarhus University（奥胡斯大学）；Griffith University（格里菲斯大学）；SciLifeLab（瑞典生命科学实验室）
University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）
A critical challenge of training deep learning models in the Digital Pathology (DP) domain is the high annotation cost by medical experts. One way to tackle this issue is via transfer learning from the natural image domain (NI), where the annotation cost is considerably cheaper. Cross-domain transfer learning from NI to DP is shown to be successful via class labels. One potential weakness of relying on class labels is the lack of spatial information, which can be obtained from spatial labels such as full pixel-wise segmentation labels and scribble labels. We demonstrate that scribble labels fr…
Unlocking the Potential of Digital Pathology: Novel Baselines for Compression
German Cancer Research Center (DKFZ)（德国癌症研究中心 (DKFZ)）；Heidelberg University（海德堡大学）；Ulm University Medical Centre（乌尔姆大学医学中心）；Technical University of Munich（慕尼黑工业大学）；University Medicine Essen（埃森大学医学中心）；Heidelberg University Hospital（海德堡大学医院）；PixelMed Publishing
